# RAG Has an Evidence Problem. Search Is Only the First Test.
**Plutonous** | November 9, 2026 | 6 min read
> Why retrieval, citation support and answer correctness need separate acceptance tests, with ALCE evidence and an operating model for document assistants.
Tags: RAG, AI Evaluation, Enterprise AI, Retrieval, Citations, Knowledge Management, AI Infrastructure, AI Strategy
---
*Cover: Generated editorial illustration of archive pages passing through a magnifying lens and a binding clamp. The artwork depicts evidence checking, not measured system behavior.*
**TL;DR: A search result, a supported citation and a correct answer are three different acceptance tests. In ALCE's 2023 human evaluation on ASQA, ChatGPT citation recall rose from 74.7% with Vanilla prompting to 79.3% with four-response reranking; that extra selection work did not establish a universal answer-accuracy gain.[2] The strategic opportunity is making evidence inspectable throughout the product, rather than treating a source link as a quality certificate.**
Retrieval-augmented generation has a persuasive sales pitch: give a language model the company's documents and make its answers useful. The original RAG paper, first released on May 22, 2020, combined a generator's learned knowledge with a retrievable Wikipedia index. Its contribution was an architecture for accessing external knowledge, not a guarantee that every sentence would become true.[1]
This is an analysis published on November 9, 2026, revisiting research released between 2020 and 2024. It does not announce a new model benchmark. The durable commercial question is how to tell whether a document assistant has supplied evidence that actually justifies its answer. The real story isn't how many files fit in the vector database. It is whether the product can expose a broken evidentiary step before a reader acts on it.
### Why This Matters Now
Document assistants increasingly serve as interfaces to operational knowledge. A retrieved policy can be relevant to the question and still be outdated, incomplete or insufficient to support the generated recommendation. Evaluation needs to identify the failing step, because replacing the generator cannot repair every retrieval or evidence problem.
## The Architecture: External Memory Is Only the Beginning
A retriever selects material; a generator uses that material to construct an answer. Those jobs can improve separately. REPLUG, first released in January 2023, treated the language model as a frozen black box and trained retrieval using feedback from its predictions. The important design implication is modularity: better document selection need not require retraining the entire generator.[4]
RETRO, first released in December 2021, explored retrieval from a database containing 2 trillion tokens. That research placed external memory inside a trained language-modeling system.[8] It is a different deployment design from attaching search to an application. Neither arrangement eliminates the need to decide which documents are authoritative, whether the corpus is current, and what the reader is permitted to see.
For buyers, this means the index is an operating asset. Its maintenance, access boundaries and version history matter alongside the model. A product that answers from yesterday's policy with today's fluent prose can look improved while becoming less dependable. The failure belongs to the information supply chain even if the generated wording is grammatically perfect.
## The Measurement: Citation Quality Has Its Own Score
ALCE explicitly evaluated fluency, correctness and citation quality as separate dimensions. Its Table 8 compared human citation judgments on ASQA. With the same ChatGPT family and task, human citation recall increased by 4.6 percentage points after reranking. The method sampled four candidate answers and selected one using an automatic citation-recall score.[2]
### Human citation recall on ASQA in ALCE (2023)
Researcher-reported human evaluation for two answer-selection strategies using ChatGPT. Citation recall measures support coverage, not overall factual accuracy.
- label: Vanilla prompting; value: 74.7; displayValue: 74.7%
- label: Four-response reranking; value: 79.3; displayValue: 79.3%
100
Citation recall
0%
100%
Source: Gao et al., ALCE, Table 8. Historical ASQA human-evaluation sample, not current ChatGPT performance. Reranking adds generation and selection work. The paper does not supply a confidence interval for this difference; the chart makes no statistical-significance claim.
The purchasing lesson is narrower than an automatic upgrade recommendation. An answer can contain a relevant citation while overstating what the passage says. Another answer can be correct yet lack adequate documentary support. If the application must justify decisions to colleagues, the second case still creates work. A quality contract should state which failure matters, who reviews it and how the product responds.
## The Control Loop: Retrieval Must Be Allowed to Fail
Corrective RAG, first released in January 2024, introduced a retrieval evaluator that assessed evidence quality and triggered different retrieval actions, including web search. The paper tested four datasets.[3] Its strategic contribution is the explicit branch: receiving documents does not oblige the system to proceed as though the evidence were sufficient.
Self-RAG, released in October 2023, trained models to retrieve on demand and critique evidence and generated text using reflection tokens.[5] That is a learned control mechanism, with its own training requirements and limitations. It should not be confused with adding the sentence “check your answer” to any prompt and assuming equivalent behavior.
A practical product can expose similar choices without claiming to reproduce either research system. It can ask for a more specific question, narrow the search, identify conflicting document versions or decline to answer a claim the corpus cannot support. These actions should be visible. Quietly continuing with weak evidence spends the reader's trust to preserve the appearance of completion.
## The Operating Model: Audit the Steps Separately
Ragas framed evaluation around relevant context, faithful use of that context and generation quality.[6] Those categories make a useful operating checklist. First inspect whether the selected passages contain the necessary information. Then inspect whether the answer follows from those passages. Finally inspect whether it addresses the actual question. Passing one check cannot be substituted for passing the others.
ARES used learned judges and a smaller human-annotated set for prediction-powered inference, seeking useful system evaluation while mitigating judge errors.[7] The implication for an evaluation budget is concrete: automate repeated checks, but reserve reviewed examples for calibration. An automated score is evidence about the system, not an independent source of truth.
Keep a versioned test set containing answerable questions, absent information, contradictory versions and requests outside a user's permissions. Log the retrieved document revision, selected passage and answer together. Reviewers should be able to locate the point at which the evidence stopped supporting the response. That record turns an opaque complaint into an actionable engineering decision.
## The Business Case: Charge for Decisions That Survive Review
A document assistant earns its place when it reduces the full cost of reaching a defensible answer. Token expenditure, retrieval charges, human review, corrections and unresolved questions all belong in that calculation. A lower model bill can coexist with a higher total operating cost if users must repeatedly reopen the originals and repair unsupported claims.
That does not mean every workplace answer needs an elaborate audit trail. The acceptance standard should follow the decision. A navigation answer can point to a document; a recommendation about a contractual obligation needs the exact passage, applicable version and clear limits. The interface should show the reader what kind of answer they received.
### A citation is a claim about support
A source link proves that a destination exists. It does not prove that the destination entails the sentence, that the source is authoritative or that a recommendation is correct. Evaluate those properties explicitly before presenting documentary support as assurance.
The strongest RAG businesses will make evidence easier to inspect and failures cheaper to resolve. Their advantage will sit in the maintained corpus, the evaluation process and the reader's ability to trace a conclusion. Search finds the material. The product must still earn the answer.
## Sources
1. [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401)
2. [Enabling Large Language Models to Generate Text with Citations](https://arxiv.org/html/2305.14627v2)
3. [Corrective Retrieval Augmented Generation](https://arxiv.org/abs/2401.15884)
4. [REPLUG: Retrieval-Augmented Black-Box Language Models](https://arxiv.org/abs/2301.12652)
5. [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://arxiv.org/abs/2310.11511)
6. [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://arxiv.org/abs/2309.15217)
7. [ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems](https://arxiv.org/abs/2311.09476)
8. [Improving language models by retrieving from trillions of tokens](https://arxiv.org/abs/2112.04426)
*Last updated: November 9, 2026*
---
*Source: [LLM Rumors](https://www.llmrumors.com/news/rag-evidence-retrieval-answer-quality)*