Retrieval-augmented generation gives language models access to information they were not trained on. It also creates a debugging problem.
Gartner’s 2026 Hype Cycle for Generative AI predicts that through 2028, at least 50% of GenAI projects will exceed their budgets because of poor architectural choices and lack of operational know-how.[1] RAG shows why architecture gets expensive – a bad answer can come from several different parts of the system.
A RAG pipeline has at least four places to fail. The corpus can contain the wrong information, retrieval can surface the wrong evidence, the model can fail to use the right evidence once it has it, or generation can turn good evidence into a bad answer. The mistake we make is evaluating such a system on one overall score.
Two systems can score similarly while needing completely different fixes. One may retrieve the right information and fail to use it. Another may retrieve poorly but still answer correctly because the model already knew enough to answer without retrieval.
Product documentation changes. Policies change. Prices change. APIs are deprecated. Permissions change. If you only evaluate the final answer, the model can cover for an upstream failure until the day it cannot.
This is the problem that RAGChecker, a framework published at NeurIPS in 2024, set out to address. Instead of treating RAG quality as a single score, it separates retrieval metrics such as claim recall and context precision from generation metrics such as context utilization, noise sensitivity, hallucination and faithfulness.[2]
That decomposition is more useful than another leaderboard number because each failure points to a different engineering intervention.
Stage one: the corpus, before anything retrieves from it
The first place I would look in a failing RAG system is the corpus. The corpus gets attention during initial setup. The harder problem is keeping it healthy afterward. Versioning, deduplication and refresh latency can quietly become problems long after the initial index is built, and it is easy to miss issues caused by staleness.
Semantic similarity alone has no preference for the current version of a fact. Suppose an internal knowledge base contains two nearly identical documents – the pricing policy from last quarter and the policy that replaced it this month. Semantically, both can be excellent matches for the same query. Without version metadata, filtering or deliberate removal of superseded content, the retrieval layer has no reason to understand that one answer is no longer authoritative.
HoH, a benchmark published at ACL 2025 specifically to study outdated information in RAG, found that stale information could significantly degrade answer accuracy and mislead models even when the correct, current information was also available.[4]
I would inspect the corpus directly. Sample the index periodically, and look for near-duplicates representing different versions of the same source. Make effective dates and version identifiers part of document metadata where they matter. Verify that superseded content is actually excluded from retrieval rather than simply left to compete with newer material.
Stage two: retrieval
Once the corpus is sound, retrieval has two basic jobs – bring back the information the answer needs and avoid surrounding it with too much information it does not.
RAGChecker captures the distinction with claim recall, which measures how much of the information needed for the answer is present in retrieved context, and context precision, which measures how much of the retrieved context is actually relevant.[2]
Splitting these out is important because a common retrieval intervention of returning more context to improve recall can hurt precision.[2]
In RAGChecker’s experiments, increasing the number of retrieved chunks from 5 to 20 raised claim recall from 61.5 to 77.6. The additional context also increased the generator’s sensitivity to noise. The same paper found a broader tradeoff – retrievers with greater coverage gave generators more useful information, but relevant chunks often carried extra material along with that information, and models did not always distinguish cleanly between the two.[2]
If the answer is missing because the relevant fact never enters the context window, increasing k, improving the embedding model, rewriting the query or changing chunk boundaries may help.
If the necessary facts are already present but buried among marginally relevant results, increasing k may make the downstream problem worse. That points instead toward ranking, filtering or a more selective retrieval strategy.
Stage three: evidence use
Good retrieval does not guarantee a good answer. Once context reaches the model, it has to decide what in that context deserves attention and what should be ignored.
The RGB benchmark, published at AAAI 2024, tested six language models on four abilities that RAG depends on – robustness to noisy documents, rejecting questions when the retrieved material does not contain the answer, integrating information spread across multiple documents, and resisting retrieved information that is factually false.[3] The models handled some retrieval noise reasonably well but had much more trouble with the other three.
On the benchmark’s negative-rejection task, where none of the supplied documents contained the answer, the highest rejection rates under the paper’s LLM-based evaluator were only 45% in English and 43.3% in Chinese.[3] Models frequently answered even when the retrieved set did not justify an answer.
Information integration exposed a different problem. In an error analysis of ChatGLM2-6B, the researchers found cases where the model merged answers to separate sub-questions, ignored one part of a compound question, or matched evidence to the wrong sub-question.[3]
When the researchers deliberately supplied false retrieved information, models could also follow it even when they previously knew the correct answer.[3]
Retrieval determines what reaches the model. The generator still has to decide what to trust, combine, ignore or reject.
RAGChecker makes the same distinction through context utilization and noise sensitivity. Its experiments found that context utilization was strongly associated with overall performance, while greater retrieval coverage could also make generators more susceptible to noise carried inside otherwise relevant chunks.[2] That creates a useful debugging boundary. If required information never entered the context, work on retrieval. If the information is clearly there but the answer omits it, miscombines it or follows nearby noise, work on how the generator consumes context.
For systems where provenance matters, I would make this observable rather than infer it from the final answer. Track which retrieved sources support which claims, and for compound questions, check whether every required part has supporting evidence. If the system cites sources, measure whether those citations actually entail the statements attached to them.
Stage four: generation
At the final stage, teams often return to a single notion of answer quality. A response can be correct but incomplete. It can be complete but unsupported by the retrieved evidence. It can be faithful to its evidence while the evidence itself is wrong, or satisfy all three and still fail the task the user was trying to accomplish. That is why generation needs more than one measure – factuality for what is wrong, completeness for what is missing, and groundedness for what the retrieved evidence actually supports. RAGChecker takes a similar approach, separating generator behavior into measures such as context utilization, noise sensitivity, hallucination and faithfulness.[2]
Keep the diagnosis useful
The point of stage-aware evaluation is to make failures easier to locate, especially in production systems where change is constant.
A useful evaluation keeps enough information to show which part of the system changed when performance moves. Knowing whether the failure came from the corpus, retrieval, evidence use or generation tells you what to do next.
That diagnosis also has to stay current. A useful evaluation suite should combine a stable regression set with a rolling sample that reflects current traffic and recently changed content. ARES shows one way to scale that evaluation: automated judges score the system while a much smaller human-annotated set is used to correct for judge errors.[5] In production, I would repeat that human calibration periodically as the system and traffic change.
A single accuracy score tells you that RAG failed. Stage-aware evaluation tells you what to fix.
