RAG Evaluation: Measure Retrieval Before You Blame the Model

Most RAG failures are diagnosed wrong. The model gets blamed for hallucinating or giving irrelevant answers, but the retrieval layer was already broken before the prompt was constructed. If you're not measuring retrieval separately, you're flying blind.

This post covers the metrics, tooling, and testing patterns you need to evaluate a RAG pipeline properly, with retrieval treated as a first-class component rather than an assumed dependency.

Why Retrieval Is the Actual Bottleneck

A language model can only work with what you give it. If the retrieved chunks are off-topic, duplicated, or missing the relevant passage entirely, no amount of prompt engineering fixes that. The model will either hallucinate a plausible answer or hedge uselessly, and both look like model failure in a casual review.

The distinction matters because the remedies are completely different. A retrieval problem might mean tuning your chunking strategy, reranking, adjusting your embedding model, or fixing your metadata filters. A generation problem might mean changing temperature, system prompt framing, or model choice. Conflating the two wastes weeks.

There is also a measurement asymmetry. Generation quality is visible to a non-technical reviewer reading outputs. Retrieval quality is invisible unless you instrument it explicitly. That invisibility is exactly why it gets missed.

What Does Good Retrieval Actually Look Like?

There are four metrics worth tracking at the retrieval layer. They come from the RAGAS framework (Shahul Es et al., 2023) and have become the practical standard.

Metric	What It Measures	Tool Support
Context Precision	Are the retrieved chunks actually relevant?	RAGAS, TruLens
Context Recall	Did retrieval surface all the relevant chunks?	RAGAS
Context Relevance	How much of each chunk is signal vs. noise?	TruLens, DeepEval
Answer Faithfulness	Does the generated answer stay within the retrieved context?	RAGAS, DeepEval

Context Precision and Context Recall have a natural tension. Raising your top-k retrieval count (say, from k=3 to k=10) typically improves recall but hurts precision because more irrelevant chunks enter the context window. You need to know where your pipeline sits on that curve before you tune anything.

Answer Faithfulness is technically a generation metric, but it belongs in the retrieval evaluation pass too, because a faithfulness failure often points back upstream. If the model is citing things not in the retrieved context, your context window is either too small or the wrong chunks are present.

Setting Up a Baseline Test Set

You need a labelled dataset before any of these metrics are computable. For most production systems, this means:

1. Sample 100 to 200 real or representative queries from your target domain.
2. For each query, manually identify which document chunks contain the ground-truth answer.
3. Record the ideal answer, or at minimum a reference passage.

This is the part people skip because it is tedious. It is also the part that makes everything else meaningful. Without ground truth, you are measuring relative change, not absolute quality, and you cannot know whether your pipeline is good enough to ship.

If you are building a legal or medical RAG system, your test set should be assembled with domain experts, not just engineers. A retrieval failure in those contexts is a correctness failure, not a UX issue.

How Do You Instrument a Live RAG Pipeline?

Offline evaluation against a test set tells you where you started. Production instrumentation tells you when things drift.

The minimal viable instrumentation setup looks like this. For every query in production, log the query text, the retrieved chunk IDs and scores, and the generated answer. You do not need to log full chunk text in every record, but you need the IDs so you can join back to your document store for analysis.

From there, you can compute a few lightweight signals without running a full RAGAS evaluation on every request:

• Retrieval score distribution: Track the similarity scores returned by your vector store (Pinecone, Weaviate, pgvector, or whatever you are using). A sudden drop in average top-1 score is an early signal that your index has drifted or a new document category is being queried.
• Empty retrieval rate: Queries that return zero chunks above your similarity threshold. This should be close to zero for a well-indexed corpus.
• Chunk diversity: If your top-k results are repeatedly pulling from the same two or three source documents, you likely have a relevance imbalance in your corpus.

Running RAGAS or DeepEval evaluations on a sampled subset of production traffic (say, 5% of queries, evaluated nightly) gives you a continuous quality signal without the cost of evaluating every request with an LLM judge.

What Breaks Retrieval That Has Nothing to Do With Embeddings?

Embedding model choice gets most of the attention, but several other factors cause retrieval failures that are harder to spot.

Chunking strategy. If you are splitting documents at fixed token boundaries (e.g., 512 tokens with 50-token overlap), you will frequently split mid-sentence or mid-concept. A question about a specific clause in a contract will retrieve a chunk that starts in the middle of that clause and ends in the middle of the next one. Semantic chunking using a sentence splitter or a document structure-aware parser (like Unstructured.io or LlamaIndex's node parsers) tends to perform better for structured documents.

Metadata filtering. In multi-tenant systems or large heterogeneous corpora, pre-filtering by metadata (date range, department, document type) dramatically reduces retrieval noise. If you are not using metadata filters and your corpus has more than a few thousand documents, you are almost certainly returning off-context chunks regularly.

Query-document mismatch. Embedding models optimise for semantic similarity, but user queries are often short and informal while source documents are long and formal. A query like "what's the refund window?" may not surface a passage that reads "customers may request a refund within 30 days of purchase." HyDE (Hypothetical Document Embeddings, Gao et al., 2022) is a practical mitigation: generate a hypothetical answer to the query, embed that instead, and retrieve against it. It adds one LLM call per query but meaningfully improves retrieval for question-answering tasks.

Index staleness. Documents updated in your source system are not automatically re-indexed. If your pipeline does not track document versions and re-embed on change, retrieval will surface outdated content with no signal to the model that the information has changed.

Choosing the Right Evaluation Framework

There are three tools worth knowing in this space.

RAGAS is the most widely adopted. It is open source, integrates with LangChain and LlamaIndex, and computes the four core metrics listed above using an LLM judge. The main limitation is cost: running a full RAGAS evaluation requires LLM calls proportional to your test set size.

DeepEval (by Confident AI) offers a broader set of metrics including G-Eval and a hallucination metric, and it supports pytest integration so you can run RAG evaluations in CI. This is useful if you want to gate deployments on retrieval quality thresholds.

TruLens is built around the concept of a "feedback function" and gives you a dashboard for tracking metrics over time. It is better suited for continuous monitoring than one-off offline evaluation.

All three support custom metrics, so if your domain has specific correctness criteria (e.g., regulatory citation accuracy), you can define a feedback function or evaluator around it.

None of these tools is a substitute for human review of sampled outputs. Use them to triage and prioritise, not to replace editorial judgment.

Conclusion

Start with a labelled test set of at least 100 queries. Measure Context Precision and Context Recall before you touch the generation layer. Set up production logging so you can track retrieval score distribution over time. Run RAGAS or DeepEval on sampled production traffic on a nightly schedule.

If your retrieval metrics are solid and your generation is still failing, then the model conversation becomes relevant. Most of the time, it will not come to that.

FAQ

What is the difference between Context Precision and Context Recall in RAG evaluation? Context Precision measures how many of your retrieved chunks are actually relevant to the query. Context Recall measures whether all the relevant chunks were retrieved. A high-precision, low-recall system gives you clean but incomplete context. A high-recall, low-precision system floods the model with noise. You need both above roughly 0.75 for a production system to behave reliably.

Do I need ground-truth labels to evaluate my RAG pipeline? For offline evaluation, yes. RAGAS and DeepEval both require reference answers or ground-truth passages to compute most metrics. For production monitoring, you can use reference-free signals like retrieval score distributions and empty retrieval rates, but these are proxies. Build a labelled test set early, even a small one, because it anchors everything else.

How does HyDE improve retrieval quality? HyDE (Hypothetical Document Embeddings) generates a plausible answer to the user's query, embeds that answer, and retrieves against it instead of embedding the raw query. Because the hypothetical answer resembles the linguistic style of your source documents more closely than a short user query does, it reduces the query-document style mismatch that hurts semantic search performance.

At what point should I consider changing my embedding model? After you have ruled out chunking, metadata filtering, and index staleness as the cause of retrieval failures. Swapping embedding models is expensive because it requires re-indexing your entire corpus. Run an A/B evaluation on your labelled test set using models like text-embedding-3-large (OpenAI), embed-english-v3.0 (Cohere), or bge-large-en-v1.5 (BAAI) before committing to a migration.

Can I run RAG evaluation in a CI/CD pipeline? Yes. DeepEval has native pytest integration, and you can define threshold-based assertions on metrics like Context Precision above 0.7 or Faithfulness above 0.8. This lets you block a deployment if a change to chunking logic or embedding configuration regresses retrieval quality. Keep the CI test set small (20 to 30 queries) to control evaluation cost and latency.

Have a project in mind? Contact Sodio Technologies to discuss your requirements and explore the right technology solution for your business.