The quality of a RAG system is determined long before the LLM generates a response, and that the four layers are knowledge ingestion, query understanding, context optimisation and grounded generation.
01Why Retrieval Quality Decides Answer Quality
The original RAG paper (Lewis et al., 2020) framed the idea: language models store facts in their parameters, but they are limited at precisely accessing and manipulating that knowledge, and at giving provenance and updating it. Pairing a model with an explicit, non-parametric memory that it retrieves from addresses all three. In practice that memory is your document collection, and the model can only be as right as the passages it is handed.
If the right passage is never retrieved, no prompt can recover it. If ten irrelevant passages are retrieved alongside it, the model may be distracted by them. So the engineering effort belongs in the pipeline that produces the context, which is why the four-layer view is useful: each layer has its own failure modes and its own fixes.
02Layer 1: Knowledge Ingestion
Ingestion turns raw documents into structured, searchable knowledge. The post lists four steps: metadata extraction, intelligent chunking, embeddings and vector storage. A few practical points for each:
- Metadata. Record source, title, section, date, document type and, where it applies, who may read it. Metadata is what lets you filter later, and access-control metadata is what keeps a RAG system from leaking documents to the wrong user.
- Chunking. Split along the document's own structure (headings, paragraphs, tables) rather than at fixed character counts where you can. Chunks that are too small lose context, and chunks that are too large dilute the match and waste the model's context window.
- Embeddings. Use the same embedding model for documents and queries, and plan for re-embedding the whole collection if you change it.
- Vector storage. Pick a store that supports the metadata filtering and hybrid search you need. See the Vector Databases page on this site.
One ingestion-time technique is worth singling out. Anthropic's Contextual Retrieval prepends a short, chunk-specific explanation of where the chunk sits in its document before embedding it (and before building the keyword index). Anthropic reports that, on its own test sets, this cut the top-20 retrieval failure rate from 5.7% to 3.7% for contextual embeddings alone (35% fewer failures) and to 2.9% when combined with contextual BM25 (49% fewer), and that adding reranking brought the reduction to 67%. Those are vendor-measured numbers on particular datasets, so treat them as a reason to test the technique on your data, not a guarantee.
03Layer 2: Query Understanding
Users do not type queries the way documents are written. This layer interprets intent before any search runs. The post names query normalisation, acronym expansion, routing and sub-query generation:
- Normalisation cleans and standardises the query: spelling, casing, stray conversational filler.
- Acronym and term expansion bridges vocabulary gaps, such as an internal abbreviation that appears spelled out in the documents.
- Routing sends the query to the right index, collection or tool, or decides that no retrieval is needed.
- Sub-query generation splits a compound question into parts that can each be retrieved for, then merges the results.
Research offers further options. HyDE (Gao et al., 2022), for example, asks a language model to write a hypothetical answer document and embeds that instead of the raw query, which can help when no labelled relevance data exists. The trade-off with all of these is that an LLM call in front of retrieval adds latency and cost, so apply them where the query actually benefits, for instance behind a quick check of whether the query is ambiguous or multi-part.
04Layer 3: Context Optimisation
The post's summary of this layer is "retrieve broadly, refine intelligently". It combines four moves:
- High-recall search. Cast a wide net first, for example the top 50 to 150 candidates, combining vector similarity with keyword search so that both meaning and exact terms (product codes, names) are caught.
- Adaptive context expansion. When a small chunk matches, bring in its neighbours or its parent section so the model sees enough surrounding text to interpret it, and expand more only when the question needs it.
- Cross-encoder reranking. A bi-encoder embeds the query and each chunk separately, which is fast enough to search a whole index. A cross-encoder reads the query and a candidate together and scores them jointly, which is more accurate but too slow to run over everything. The standard pattern, described in the Sentence Transformers documentation as retrieve and re-rank, uses the bi-encoder to fetch candidates and the cross-encoder to reorder a short list.
- Metadata filtering. Narrow by date, source, language or permissions, ideally inside the search itself so the filter does not discard your best results after the fact.
Fewer, better passages beat more passages. The "Lost in the Middle" study found that language models often perform best when the relevant information is at the start or end of a long input and significantly worse when it is in the middle, so a short, well-ordered context is safer than a long one.
05Layer 4: Grounded Generation
The last layer is where the model writes the answer, and its job is to stay inside the evidence. The post pairs optimised context with citations and guardrails to improve trust, reduce hallucinations and meet enterprise reliability expectations. In practice that means:
- Instruct the model to answer only from the supplied context and to say so when the context is insufficient.
- Ask for citations that point to specific retrieved passages, and show them to the user so answers can be checked.
- Apply guardrails on both sides of the model: screen inputs and check outputs for unsupported claims, sensitive data and policy violations.
Grounding reduces hallucination but does not eliminate it; a model can still misread a correct passage. The Hallucination Mitigation and Guardrails pages on this site cover those techniques.
06The Pattern: Recall, Expand, Rerank, Ground
The post calls the winning retrieval pattern for 2026: high-recall search, then context expansion, then cross-encoder reranking, then grounded LLM generation. That is the author's view rather than a measured standard, but it is a sound default because each stage handles a different weakness: the first makes sure the right passage is somewhere in the candidates, the second makes sure it is readable in context, the third pushes it to the top, and the fourth makes the model use it faithfully.
Where the time goes
Reranking and query rewriting both add model calls to every request. Budget for them: rerank a short list rather than hundreds of candidates, cache embeddings and frequent queries, and skip the optional steps for simple queries. The post's goals of lower latency and lower inference cost are in direct tension with higher recall and precision, which is why measurement matters.
07Measuring It: Recall, Precision, Latency, Cost
The post sets four objectives: higher recall, higher precision, lower latency and lower inference costs. Evaluate retrieval separately from generation so you can tell which one is failing:
| Objective | What to measure | Levers |
|---|---|---|
| Recall | How often the needed passage appears in the retrieved candidates | Hybrid search, contextual chunks, wider candidate pool, query expansion |
| Precision | How much of the final context is relevant | Reranking, metadata filters, smaller final context |
| Latency | Time per stage and end to end | Fewer LLM calls, rerank a short list, caching, parallel retrieval |
| Cost | Tokens and model calls per answer | Shorter context, cheaper models for rewriting and reranking |
For answer quality, the Ragas framework offers reference-free evaluation of RAG pipelines across dimensions such as how relevant and focused the retrieved passages are and whether the generation is faithful to them. Pair automated scores with a fixed set of real questions that you re-run whenever you change chunking, embeddings, the reranker or the prompt.
08Putting It Together
An efficient RAG system is built backwards from the answer. Ingest knowledge with structure and metadata, interpret the query before searching, retrieve widely and then refine to a small, well-ordered context, and make the model answer from that context with citations and guardrails. As the post puts it, competitive advantage comes less from choosing a larger model than from designing a smarter retrieval architecture.
Further reading and grounding for the details referenced above:
- Manas Dasgupta's LinkedIn post on RAG architecture — the four-layer outline this article expands.
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the original RAG paper.
- Introducing Contextual Retrieval (Anthropic) — the chunk-context technique and its reported results.
- Retrieve & Re-Rank (Sentence Transformers) — bi-encoder retrieval followed by cross-encoder reranking.
- Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE).
- Lost in the Middle: How Language Models Use Long Contexts.
- Ragas: Automated Evaluation of Retrieval Augmented Generation.
- On this site: RAG Architectures Elucidated, Vector Databases, Hallucination Mitigation, Guardrails for Agentic AI and GraphRAG Architectural Patterns.