A model only knows what it was trained on, plus whatever fits in its context. To ask questions about your own files, you do not retrain anything. You use retrieval-augmented generation (RAG): find the passages that matter, then give them to the model with your question.
How it works
- Split your documents into chunks of a few paragraphs.
- Embed each chunk: an embedding model turns its meaning into a list of numbers.
- Store those vectors in a small database.
- Retrieve: embed your question the same way and pull the chunks whose vectors are closest.
- Generate: the chat model reads those chunks and your question, and answers from them.
Two models are involved. A small embedding model (a few hundred megabytes) handles steps 2 and 4, and your chat model handles step 5. Embeddings are specific to a model: if you switch embedding models you must re-embed everything.
Apps that do it for you
AnythingLLM is a desktop app with separate workspaces and visible sources. Open WebUI adds document chat to Ollama or any local API. Khoj searches your notes. For your own pipelines, embedding servers such as Text Embeddings Inference and libraries such as Sentence Transformers run the embedding model.
Settings and mistakes that matter
- Context length. The retrieved chunks must fit in the model’s context. If an app silently defaults to a small window, retrieval looks broken. Raise it (cost).
- Chunk size. A thousand or so characters with a little overlap is a common starting point. Too large mixes topics; too small loses context.
- How many chunks. Four to six is typical. More can distract the model rather than help.
- Messy source files. Scanned PDFs, tables and repeated headers extract badly. Clean the text, or run OCR, first.
- Always check the sources. A good app shows which passages the answer came from. If it cannot, treat the answer with suspicion.
Models tagged RAG are the ones that stick closest to the supplied text. Your documents stay on your computer throughout (privacy).