How were data extracted from PDF documents for the RAG system?

To extract data from PDF documents in RAG (Retrieval-Augmented Generation) systems, the following approach is usually used:

1. Parsing PDFs: Use libraries like PyMuPDF, pdfplumber, or Apache PDFBox to extract text and structured data from PDFs.

2. Preprocessing text: Clean the text from unnecessary characters, normalize, and remove stop words.

3. Segmenting into fragments: Divide the text into logical blocks (paragraphs, pages, sentences) for easier searching.

4. Vectorization: Convert each fragment into a vector using embedding models (e.g., Sentence Transformers).

5. Indexing: Store the vectors in a database for quick search (e.g., FAISS).

6. Searching and generating: When a query is made, the system searches for relevant fragments and uses them to generate a response.

Example Python code for extracting text from a PDF using pdfplumber:

import pdfplumber

with pdfplumber.open("document.pdf") as pdf:
    full_text = "".join(page.extract_text() for page in pdf.pages)

print(full_text)

Then, this text can be segmented and processed for the RAG system.