RAG pipeline: from documents to grounded answers

What's actually being asked

Give the complete step-by-step RAG workflow — every stage from raw documents to a final grounded answer — not just "retrieval + generation" at a high level.

Answer

Why RAG exists, in one line

An LLM's parametric knowledge is frozen at training time and can't cite anything; RAG fixes both problems by retrieving relevant text at query time and feeding it into the prompt as grounding context, so the model answers from that text instead of purely from memory. The flow below matches the diagram.

1. Document ingestion

Collect the raw source documents (PDFs, web pages, database records, internal wikis), parse them into clean plain text, and strip out noise (headers/footers, boilerplate). Useful metadata — source, title, date, section — is extracted here too, since it'll travel with every chunk downstream.

2. Chunking

Split each document into smaller passages — fixed-size windows with some overlap (e.g., ~300 tokens with ~50 overlap), or splitting along natural boundaries like paragraphs/sections. This is necessary because (a) embedding models and the LLM's context window both have length limits, and (b) retrieval precision is better over focused chunks than over whole documents — a 50-page PDF returned whole would drown the one relevant paragraph in noise.

3. Embedding generation

Each chunk is passed through an embedding model to get a dense vector that captures its semantic meaning — semantically similar chunks end up close together in vector space, even if they don't share exact words.

4. Indexing

All chunk vectors (plus the chunk text and metadata) are stored in a vector index/database built for fast approximate-nearest-neighbor search (e.g., FAISS-style structures), so that at query time the system doesn't have to compare against every chunk linearly.

5. Retrieval

At query time, the user's question is embedded with the same embedding model used for the corpus (this matters — mismatched embedding spaces give meaningless similarity scores). A similarity search — commonly cosine similarity, sim(q,c) = (q·c)/(‖q‖‖c‖) — finds the top-k most relevant chunks in the index.

6. Prompt augmentation

The retrieved chunks are inserted into the LLM prompt alongside the original question and an instruction, e.g.:

Using only the following context, answer the question. If the answer isn't
in the context, say you don't know.
 
Context:
[chunk 1 text]
[chunk 2 text]
 
Question: {user question}

7. Generation

The LLM generates its answer conditioned on this augmented prompt — so the answer is grounded in the retrieved text rather than relying solely on whatever the model memorized during pretraining.

8. Source grounding

The system keeps track of which chunk(s)/document id(s) actually fed the answer, so the final response can cite them (or at least be traced back to them). This is what makes RAG answers verifiable and meaningfully reduces (though doesn't eliminate) hallucination, since a wrong answer can be checked against the exact source it claims to come from.

Worked example

Corpus: a short article on Transformers, chunked into 3 passages (intro, self-attention mechanics, applications). User asks "What does the √d_k term do in attention?" → query embedded → top-1 retrieved chunk is the self-attention passage → prompt = that chunk + the question → LLM answers "it scales the dot products down so softmax doesn't saturate into tiny gradients" — grounded in, and traceable to, that one chunk.

Exam tip

Two things examiners probe: (1) the query must be embedded with the same model as the corpus — a common "what's wrong with this RAG setup" trap is a mismatched embedding model; (2) RAG is an inference-time technique — it does not fine-tune the LLM's weights, it just changes what's in the prompt.