TL;DR RAG, retrieval-augmented generation, is three steps: find the handful of passages in your documents that relate to the question, paste them into the prompt, and let the model answer from them. It is how you make a model answer over knowledge it was never trained on, without retraining anything.
A model only knows what it saw in training. It has never read your company handbook, last week's support tickets, or the contract you signed this morning. Ask it about any of those and it will either admit it does not know or, worse, invent something plausible. RAG is the standard fix, and despite the intimidating name the whole idea fits in one sentence: before you ask the model the question, go and fetch the relevant text, and hand it over along with the question.
The problem it solves
You might think the answer is to paste your entire knowledge base into every prompt. But as I covered in the tokens piece, the context window is a fixed budget, and a real document store is far too big to fit. Even if it fit, you would pay for all of it on every single question, and the model's accuracy drops when the useful sentence is buried in thousands of irrelevant ones.
So the goal is not to give the model everything. It is to give the model only the few passages that actually bear on this question. That selection step is the “retrieval” in retrieval-augmented generation, and it is where the embeddings from the last piece do their work.
The three steps
Retrieve. Ahead of time, you split your documents into passages and embed each one, turning it into a point in the meaning-space from the embeddings piece. When a question arrives, you embed it too and keep the handful of stored passages sitting nearest to it. Those are the passages most likely to contain the answer, chosen by meaning rather than by matching keywords.
Augment. You build a single prompt that stitches those passages together with the user's question and a short instruction: answer using the context below, and if it is not there, say so. This is the step the name is really about. You are augmenting the prompt with fetched knowledge.
Generate. The model answers as it always does, except now the facts it needs are sitting right there in the prompt. The answer is grounded in your documents, and because you know which passages you supplied, you can cite them, which is how RAG systems show their sources.
Why not just fine-tune the model?
The common alternative is fine-tuning: retraining the model on your data so the knowledge lives inside its weights. For most “answer over my documents” problems, RAG wins, and the reasons are practical. Your documents change; with RAG you just update the store, with no retraining. RAG can cite the exact passage it used, so an answer is checkable. And when the passage is not found, a good RAG setup can say “I do not know” instead of confidently making something up, because you told it to answer only from the context.
Fine-tuning still has its place, mostly for teaching a model a style, a format, or a narrow skill, rather than facts. But it is the wrong tool for “know the current contents of this folder.” Facts that change want retrieval, not retraining.
Why any of this matters
Almost every “chat with your documents” product you have seen is RAG underneath. Once you can see the three steps, these systems stop looking clever and start looking like plumbing you can reason about, which is exactly what you want when one goes wrong. And the failures live in specific steps: a bad answer is usually a retrieval problem, the right passage never made it into the prompt, far more often than it is the model failing to reason.
That is the framing I keep. RAG is not a kind of model or a product; it is a pattern, retrieve then generate. Get the retrieval right and the generation mostly takes care of itself, which is why the unglamorous question, how you split and search your documents, is where I spend most of my time.