How were documents uploaded to the vector database? How was chunking performed?

Uploading documents to a vector database typically involves several steps:

1. Preprocessing the text — cleaning, normalization, removing stop words.
2. Chunking — dividing the document into logical parts (chunks), such as paragraphs, sentences, or fixed sizes (e.g., 500 tokens). This helps preserve context and improve search quality.
3. Vectorization — converting each chunk into a vector using embedding models (e.g., BERT, Sentence Transformers).
4. Indexing — storing the vectors in a vector database for fast retrieval.

Example of chunking in Python using the NLTK library:

import nltk
nltk.download('punkt')
from nltk.tokenize import sent_tokenize

def chunk_text(text, max_sentences=5):
    sentences = sent_tokenize(text)
    chunks = []
    for i in range(0, len(sentences), max_sentences):
        chunk = ' '.join(sentences[i:i+max_sentences])
        chunks.append(chunk)
    return chunks

# Example usage
text = "Your large text here..."
chunks = chunk_text(text)

After that, each chunk is passed to an embedding model to obtain a vector, which is then stored in the database.