RAG on a VDS: AI Answers Based on Your Documents

What Is RAG?

RAG (Retrieval-Augmented Generation) is a technique in which an LLM first searches a knowledge base for relevant fragments before generating a response. This lets the model answer questions about your documents accurately, without requiring fine-tuning.

Components of a RAG System

• Document loaders — PDF, DOCX, TXT, web pages
• Text Splitter — splitting into chunks (500-1000 tokens)
• Embedding model — converting text into vectors
• Vector database — Chroma, Qdrant, FAISS
• LLM — generating a response based on the retrieved context

Installing Dependencies

pip install langchain langchain-ollama langchain-community
pip install chromadb pypdf sentence-transformers

Complete RAG Example in Python

from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import OllamaEmbeddings
from langchain_ollama import OllamaLLM
from langchain.chains import RetrievalQA

# Load and split the document
loader = PyPDFLoader("document.pdf")
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)

# Create the vector database
embeddings = OllamaEmbeddings(model="nomic-embed-text")
db = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")

# RAG chain
llm = OllamaLLM(model="llama3.2")
qa = RetrievalQA.from_chain_type(llm=llm, retriever=db.as_retriever())
result = qa.invoke("What is the main topic of this document?")
print(result["result"])

Embedding Models for Ollama

# Download a specialized embedding model
ollama pull nomic-embed-text

# Or the English variant
ollama pull mxbai-embed-large

Common RAG Errors

Most RAG problems are not in the code but in retrieval quality — the model answers without using the context because the wrong context was retrieved in the first place.

The Model Ignores the Retrieved Context

A common cause is a prompt that does not force the model to rely specifically on the retrieved chunks: it answers from general knowledge unless you explicitly say "answer only based on the provided context, and if the answer is not there, say you do not know." The second cause is a retriever returning too few chunks (k=1-2), so the needed fragment never makes it into the selection.

Dimension Mismatch When Adding Documents to Chroma

chromadb.errors.InvalidDimensionException: Embedding dimension 768 does not match collection dimensionality 1024

This error occurs when the same Chroma collection received embeddings generated by different models — they have different vector dimensions. The fix is to use a single embedding model for the entire lifetime of the collection, or create a separate collection when switching models.

Chroma, Qdrant, and FAISS Compared

When RAG Is Not a Good Fit

• There are few documents and they all fit within the model's context window — it is simpler to pass the whole text in the prompt, without a vector database.
• Data changes almost in real time — you would have to constantly re-index chunks, and the answer would still lag behind.
• You need exact calculations or aggregation across all documents at once — RAG finds similar fragments, it does not sum values or compare every record.
• User questions require reasoning across the whole knowledge base rather than individual fragments — full-text analysis or a SQL query fits better than vector search here.

Verifying Search Quality

results = db.similarity_search_with_score("a test question from your knowledge base", k=3)
for doc, score in results:
    print(score, doc.page_content[:100])

Run a search with a question you know is answered somewhere in the documents, and check that the right chunk ends up among the most relevant results. If it is in the database but does not make the top-k, try reducing the chunk size or increasing k before switching embedding models.

Another nuance is updating documents in the database. If the source file changes, simply re-adding chunks to the same Chroma collection creates duplicates instead of replacing the old versions. Before re-indexing, either delete the chunks tied to the previous source identifier or create a separate collection for each document version and explicitly point the retriever at the current one.