Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA

Paper: 2603.24580 Authors: Saahil Mathur, Ryan David Rittner, Vedant Ajit Thakur, Daniel Stuart Schiff, Tunazzina Islam Categories: cs.CL, cs.AI, cs.CY, cs.IR, cs.LG

Abstract

The paper studies retrieval-augmented generation for AI policy analysis using the AGORA corpus of 947 policy documents. It combines a ColBERT retriever fine-tuned with contrastive learning and a generator tuned with DPO. The authors find that better retrieval scores do not reliably lead to better answers.

Key Contributions

• Targets AI governance and policy question answering
• Uses the AGORA corpus of 947 curated AI policy documents
• Combines a ColBERT retriever with a DPO-aligned generator
• Uses synthetic queries and pairwise preferences to adapt the system to the policy domain
• Shows that retrieval gains do not consistently translate into better end-to-end answer quality

Methodology

The system is built as a standard RAG pipeline but tuned for policy QA rather than open-domain retrieval. A retriever is trained to surface relevant policy documents, while the generator is optimized to answer using the retrieved evidence. The authors also use synthetic queries and preference data to make both parts more domain aware.

The interesting point is that retrieval is only one piece of the chain. Even if the retriever gets better at finding related documents, the generator can still miss the right evidence, overstate confidence, or hallucinate when the evidence is weak. The paper uses that tension to question a common assumption in RAG systems.

Results

Retrieval metrics improve after fine-tuning, but the end-to-end QA quality does not consistently follow. In some settings, the system still produces confident answers without enough supporting evidence. That makes the study especially relevant for policy and governance use cases, where being confidently wrong is worse than being unsure.

The paper’s main lesson is that retrieval quality alone is not a sufficient proxy for answer quality. Evaluation has to measure the whole pipeline.

Takeaways

1. Better retrieval does not automatically mean better answers
2. End-to-end QA evaluation matters more than isolated retriever metrics
3. Domain-specific RAG for policy work needs evidence-sensitive generation
4. Synthetic queries and preference tuning help, but do not solve the full problem
5. Confident hallucinations remain a risk when the retrieved evidence is weak