Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA

TL;DR

This study investigates how improvements in retrieval components of RAG systems do not always lead to better, more reliable answers in complex AI policy domains, highlighting the need for holistic system evaluation.

Contribution

The paper demonstrates that enhancing retrieval quality alone does not guarantee improved answer accuracy in policy-focused RAG systems, emphasizing the importance of comprehensive evaluation.

Findings

Domain-specific fine-tuning improves retrieval metrics.

Stronger retrieval can increase hallucinations when relevant documents are missing.

Component improvements do not necessarily enhance end-to-end answer quality.

Abstract

Retrieval-augmented generation (RAG) systems are increasingly used to analyze complex policy documents, but achieving sufficient reliability for expert usage remains challenging in domains characterized by dense legal language and evolving, overlapping regulatory frameworks. We study the application of RAG to AI governance and policy analysis using the AI Governance and Regulatory Archive (AGORA) corpus, a curated collection of 947 AI policy documents. Our system combines a ColBERT-based retriever fine-tuned with contrastive learning and a generator aligned to human preferences using Direct Preference Optimization (DPO). We construct synthetic queries and collect pairwise preferences to adapt the system to the policy domain. Through experiments evaluating retrieval quality, answer relevance, and faithfulness, we find that domain-specific fine-tuning improves retrieval metrics but does not consistently improve end-to-end question answering performance. In some cases, stronger retrieval counterintuitively leads to more confident hallucinations when relevant documents are absent from the corpus. These results highlight a key concern for those building policy-focused RAG systems: improvements to individual components do not necessarily translate to more reliable answers. Our findings provide practical insights for designing grounded question-answering systems over dynamic regulatory corpora.