A RAG system can produce a wrong answer for many different reasons.
The correct document may not be retrieved. It may rank too low. A chunk may omit a critical sentence. A filter may select an outdated policy. A reranker may promote irrelevant evidence. Or the LLM may receive the right evidence and still invent a conclusion.
That is why RAG evaluation cannot be reduced to:
A better framework asks:
The practical chain is:
Three levels of RAG evaluation
| Level | Main question |
|---|---|
| Retrieval | Did we retrieve the right evidence? |
| Generation | Did the model use evidence correctly? |
| End-to-end | Did the system solve the task? |
If retrieval fails, prompt engineering cannot recover information the model never received.
Why final-answer accuracy is not enough
A wrong answer can come from a retrieval miss, ranking failure, stale source, or generation failure.
Each requires a different fix.
Retrieval metrics
Recall@K
Did the top K contain the evidence required to answer?
Precision@K
How much retrieved material was actually relevant?
MRR
How highly ranked was the first relevant result?
NDCG
How well were results ordered when relevance has multiple levels?
Ground truth vs no ground truth
When relevance labels exist, use deterministic ranking metrics.
When they do not, use human or model-based relevance judgments while gradually building labeled ground truth.
Evaluate final context
Retrieval candidates and final model context are not always the same.
A system may retrieve 50, filter, rerank, compress, and send only eight chunks.
Evaluate what actually reaches the model.
Groundedness
Groundedness asks:
A response can be fluent and relevant but still contain unsupported claims.
Groundedness vs correctness
A response can faithfully repeat an outdated document. It is grounded but incorrect relative to current truth.
Evaluate both.
Relevance and completeness
A grounded response may still fail to answer the question or may omit a required condition.
Measure relevance, completeness, correctness, and groundedness separately.
Citation evaluation
Measure citation correctness, completeness, authority, and location.
A fabricated citation can create false trust.
Evaluate chunking
Test multiple chunk sizes and strategies, then compare Recall@K, Precision@K, context tokens, answer correctness, latency, and cost.
Evaluate overlap
Overlap can preserve boundary information but increases duplicate vectors and context. Measure whether it helps.
Evaluate embedding models
Public benchmarks may not match your domain. Compare embedding models using domain-specific query-document pairs.
Dense, sparse, and hybrid retrieval
Test dense only, sparse only, and multiple hybrid weights.
Hybrid retrieval is often useful when semantic concepts and exact identifiers both matter.
Evaluate reranking
A reranker can improve final evidence quality at extra cost and latency. Measure whether the benefit justifies the overhead.
Evaluate Top-K
Too small may miss evidence. Too large may introduce noise.
Choose the minimum K that provides sufficient evidence.
Evaluate metadata filters
Benchmarks without realistic tenant, region, date, permission, product, or jurisdiction filters can be misleading.
Measure freshness
Track:
This is crucial for policy, regulation, pricing, inventory, and customer records.
Evaluate temporal correctness
The newest document is not always correct for a historical question. Include historical and future-effective cases.
Evaluate permission correctness
Unauthorized information should not appear in retrieval candidates.
Authorization belongs before the model.
Build a RAG golden dataset
Useful fields include query, expected answer, relevant documents, required facts, forbidden sources, expected citations, and segment.
This supports retrieval, ranking, answer, citation, freshness, and security evaluation.
Include hard negatives
Use plausible but wrong documents such as old policies, similar products, adjacent regulations, or another tenant's data.
Slice results
Aggregate metrics can hide failures. Slice by document type, language, product, jurisdiction, date, table vs prose, and scanned vs digital content.
Evaluate tables separately
Measure extraction, row/column preservation, retrieval, and numeric answer correctness.
Structured-data RAG
For SQL, graph, API, or data-product retrieval, evaluate field selection, query correctness, filters, row completeness, and computation accuracy.
Use deterministic checks when possible
Examples include correct document retrieved, correct limit in answer, valid citation ID, unauthorized document absent, and valid output schema.
Use judges for more subjective dimensions.
Use LLM judges carefully
LLM judges are useful for relevance, completeness, and faithfulness, but should be calibrated against human-reviewed examples.
Measure latency by stage
Break out query transformation, retrieval, reranking, context assembly, generation, and validation.
Optimize the actual bottleneck.
Measure cost per successful answer
A cheap query can be expensive overall if it fails often and requires human intervention.
Use:
Build a RAG scorecard
Useful dimensions include Recall@5, NDCG@10, grounded answer rate, correctness, citation correctness, permission leakage, freshness SLA, P95 latency, and cost per successful answer.
Treat RAG configuration as knobs
Important variables include parser, chunk strategy, chunk size, overlap, embedding model, index, similarity metric, hybrid weight, Top-K, reranker, filters, prompt, LLM, and grounding threshold.
Run controlled experiments
Compare configurations on the same dataset and measure:
Regression test before changes
Embedding, chunking, retrieval, reranking, prompt, model, and metadata changes can all alter behavior.
Continuous monitoring
Track unanswered questions, low-grounding responses, retrieval misses, user corrections, citation clicks, escalations, stale-source incidents, latency, and cost.
When an important production failure appears, add it to the evaluation set.
RAGOps loop
A mature loop is:
Bottom line
Do not evaluate RAG with one final-answer score.
Evaluate the evidence chain:
- Did retrieval find the right information?
- Was it ranked and filtered correctly?
- Did the model stay grounded?
- Was the answer correct, complete, and relevant?
- Did the system remain within latency, cost, freshness, and security limits?
Then identify which configuration variable caused a failure and measure which change improves the outcome.