The Problem RAG Solves
A language model knows only what was in its training data, frozen at a cutoff date. It does not know your internal documentation, your product catalogue, your policies or anything that happened after training.
Ask it about those and it will either say it does not know, or — worse — generate something plausible and wrong.
RAG solves this by retrieving relevant material at the moment of the question and placing it in the model's context, so the answer is grounded in documents you control.
How It Works
Two phases.
Indexing, done in advance: documents are split into chunks of a few hundred tokens, each chunk is converted to an embedding — a numerical vector representing its meaning — and the vectors are stored in a vector database.
Retrieval and generation, at query time: the user question is embedded using the same model, the database returns the chunks whose vectors are closest to it, those chunks are inserted into the prompt, and the model generates an answer from them.
The model is not modified. It is simply given better context.
Why Retrieval Quality Decides Everything
RAG output can only be as good as what retrieval finds. If the relevant passage is not returned, the model cannot use it, and generation quality cannot compensate.
Most disappointing RAG systems have a retrieval problem rather than a model problem. Common causes: chunks split badly so context is severed mid-argument, chunks too large so the relevant sentence is diluted, an embedding model poorly suited to the domain, and no handling for questions whose answer spans several documents.
Improvements that usually help: hybrid search combining semantic and keyword matching, reranking retrieved results before generation, and chunking on semantic boundaries rather than fixed token counts.
RAG Versus Fine-Tuning
They solve different problems and are frequently confused.
RAG gives the model access to information. Use it when the model needs to know facts it was not trained on, when that information changes, or when you need citations back to sources.
Fine-tuning changes how the model behaves — tone, format, domain vocabulary, task-specific patterns. Use it when the model knows enough but does not respond the way you need.
Wanting the model to answer questions about your documents is a RAG problem. Wanting it to write in your house style is a fine-tuning problem. Many production systems use both.
What RAG Does Not Fix
RAG reduces hallucination substantially; it does not eliminate it. A model can still misread a retrieved passage or blend it with training knowledge incorrectly.
It also cannot answer questions requiring reasoning across the whole corpus — "what are the three most common themes across these 400 documents" is not a retrieval problem, because no small set of chunks contains the answer.
And it inherits the quality of your documents. RAG over outdated or contradictory internal documentation produces confidently outdated or contradictory answers.
Diagnosing a RAG System That Answers Badly
When a retrieval system gives a poor answer there are three distinct failure points, and fixing the wrong one is the usual outcome.
The document was never retrieved. Check first, always: search the index manually for the passage that should have answered it. If it is not in the results, the problem is retrieval — chunking, embedding, or the query — and no prompt change will fix it.
It was retrieved and ignored. The passage is in the context and the answer contradicts it. This is a prompting or model problem, and often a position problem — material in the middle of a long context is used less reliably than material at either end.
It was retrieved, used, and the source was wrong. The system worked perfectly and the corpus was stale. This is the most dangerous failure because it is invisible — the answer is confident, well-sourced and wrong, and the citation makes it more believable.
Always check in that order. Teams routinely rewrite prompts for a retrieval problem and re-tune retrieval for a stale-document problem.
The Corpus Is the Product
Most RAG effort goes into the pipeline and most of the quality comes from the documents. The ratio should be closer to the reverse.
Decide what is authoritative and index only that. Indexing everything means indexing three superseded versions of the policy, and the system has no way to prefer the current one.
Remove rather than add. The highest-return maintenance in most deployments is deleting outdated documents, and it is never anyone's job.
Date everything and surface the date in the answer, so a reader can judge currency themselves.
Cite to a location a person can open. A citation that cannot be checked is decoration, and it increases trust without increasing reliability — the worst combination.
And close the loop. When a user reports a bad answer, the question is which of the three failure points it was, and the answer should change either the corpus or the pipeline. A system with no feedback path degrades silently as the documents age around it.