The Retriever article argued that a retriever’s accuracy should be measured independently of the language model’s generation quality. This article covers exactly how that measurement actually gets done: retrieval evaluation.
The simple definition
Retrieval evaluation measures how well a retriever finds the genuinely relevant chunks for a given query, independent of what the language model does with those chunks afterward. Recall from the Evaluation Basics phase’s Precision and Recall articles — retrieval evaluation directly reuses these same core metrics, applied specifically to the question of whether retrieval found the right content, not whether the final generated answer was good.
Why this needs to be measured separately from the final answer
Recall from the Retriever article’s real, practical insight: a RAG system’s final answer can fail for two genuinely different reasons — bad retrieval (the wrong chunks were found) or bad generation (good chunks were found, but the model used them poorly).
Without measuring retrieval on its own, these two failure modes get hopelessly tangled together, making it impossible to know which part of the system actually needs fixing when something goes wrong.
flowchart LR
A[Test query] --> B[Retriever returns chunks]
B --> C{Compare against known-relevant chunks}
C --> D[Precision: what fraction of returned chunks were actually relevant?]
C --> E[Recall: what fraction of all relevant chunks were actually found?]How this actually gets measured, step by step
This connects directly to the mechanics covered in the Precision and Recall articles, applied here specifically. Building a retrieval evaluation set requires a collection of test queries, each paired with a known set of genuinely relevant chunks — established either by human judgment or by careful, deliberate test-set construction.
For each test query, the retriever’s actual returned results get compared against that known-relevant set: precision measures what fraction of returned chunks were genuinely relevant, and recall measures what fraction of all genuinely relevant chunks in the knowledge base actually got returned.
ANALOGY vs. TECHNICAL REALITY
Analogy: Think of testing a librarian’s research skills by giving them a set of test questions where you already know, in advance, exactly which specific books would be genuinely useful answers — then checking how many of the books they actually pulled were genuinely useful, and how many of the genuinely useful books they missed entirely.
Where this breaks down: Testing a librarian involves a human judge assessing a small number of test cases individually. Retrieval evaluation is typically run automatically, at scale, across potentially hundreds of test queries, recalculating precision and recall metrics every time a chunking strategy, embedding model, or retrieval technique changes — a repeatable, quantitative process rather than a one-off assessment.
A specific metric worth knowing: NDCG
It’s worth naming one additional, real metric commonly used specifically for retrieval, beyond plain precision and recall.
NDCG (Normalized Discounted Cumulative Gain) accounts for the fact that ranking order matters, not just whether relevant chunks were returned at all — a relevant chunk ranked 1st is more valuable than the same relevant chunk ranked 15th, since a language model’s context window and attention, as covered throughout the Transformers phase, don’t necessarily weigh every retrieved chunk equally.
NDCG specifically rewards retrieval systems for putting the most relevant results first, not just for including them somewhere in the results.
A concrete example, layered
For a simple beginner example: testing a company’s customer-support retriever against 50 known test questions, each paired with the specific FAQ entry that should be retrieved, lets a team calculate exactly what percentage of the time the retriever actually surfaced that correct entry in its top results.
For a production example: teams building serious RAG products commonly maintain a dedicated retrieval evaluation dataset, re-running it automatically every time they experiment with a new chunk size, embedding model, or reranker — treating retrieval quality as a genuinely measurable, trackable engineering metric, exactly the kind of disciplined evaluation process first established throughout the Evaluation Basics phase.
Why skipping this step is a genuinely common, costly mistake
It’s worth being direct about a real, practical risk here. Without dedicated retrieval evaluation, teams can only judge their RAG system by its final, generated answers — which, as covered throughout this phase, conflates retrieval quality with generation quality and makes it genuinely difficult to diagnose exactly what to fix when results are disappointing.
Dedicated retrieval evaluation is precisely what makes systematic, targeted improvement possible, rather than guessing at which component actually needs attention.
Calculate retrieval metrics on one query
Suppose three chunks are known to be relevant: A, C, and F. The retriever returns five results in this order:
Rank: 1 2 3 4 5
Returned: A B C D E
Relevant: ✓ ✓Precision@5 = 2 relevant retrieved / 5 retrieved = 0.40
Recall@5 = 2 relevant retrieved / 3 relevant total = 0.67
MRR = 1 / rank of first relevant result = 1 / 1 = 1.00Precision asks how clean the returned list is. Recall asks how much of the needed evidence was found. MRR emphasizes how early the first relevant result appears.
Build a useful evaluation set
Each test case should contain a realistic query, known relevant chunks or documents, metadata and permission expectations, and difficult negatives that look plausible but are wrong. Include exact identifiers, paraphrases, vague follow-ups, multiple languages, outdated policies, and questions with no answer in the knowledge base.
Offline metrics should be complemented by production signals such as zero-result rates, click or citation behavior, latency, and carefully reviewed failure samples.
A realistic evaluation table
Suppose a team prepares 100 employee questions and records the correct supporting document for each. Retriever A places a correct passage in its top 5 for 82 questions, so Recall@5 is 82/100 = 82%. After a chunking change, Retriever B succeeds for 89 questions, so Recall@5 becomes 89%.
Those are clearly marked worked numbers, not a Google benchmark. The ground truth is the human-verified question-to-source mapping stored in the evaluation dataset. Teams should also inspect difficult misses, because a seven-point gain does not show whether failures affect harmless FAQs or important compliance questions.
Common misconception
A frequent beginner assumption: that if a RAG system’s final answers seem good, retrieval must also be working well, making separate retrieval evaluation unnecessary.
This isn’t reliable — a capable language model can sometimes produce a reasonable-sounding answer even from mediocre retrieved context, masking a genuine retrieval problem that would surface clearly on a harder question; dedicated retrieval evaluation catches this kind of hidden weakness that final-answer quality alone can miss.
Where this fits in what comes next
You now understand measuring retrieval quality in isolation. The next article, RAG Evaluation, covers the complementary, broader measurement — assessing the complete, end-to-end system, including how well the language model actually uses whatever retrieval provides it.
In one sentence
Retrieval evaluation measures a retriever’s accuracy in isolation, reusing precision, recall, and ranking-aware metrics like NDCG from the Evaluation Basics phase, and it’s precisely this separate measurement that lets a team diagnose whether a disappointing RAG system needs better retrieval or better generation, rather than guessing.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed