RAG-X: Systematic Diagnosis of Retrieval-Augmented Generation for Medical Question Answering

TL;DR

RAG-X is a diagnostic framework that evaluates retrieval and generation components separately in medical QA systems, revealing hidden failure modes and improving transparency for clinical AI applications.

Contribution

It introduces a novel diagnostic approach with CUE metrics to independently assess retriever and generator performance in complex medical QA tasks.

Findings

Identifies a 14% gap between perceived success and evidence-based grounding.

Reveals hidden failure modes in retrieval-augmented generation systems.

Provides a framework for targeted improvements in clinical AI systems.

Abstract

Automated question-answering (QA) systems increasingly rely on retrieval-augmented generation (RAG) to ground large language models (LLMs) in authoritative medical knowledge, ensuring clinical accuracy and patient safety in Artificial Intelligence (AI) applications for healthcare. Despite progress in RAG evaluation, current benchmarks focus only on simple multiple-choice QA tasks and employ metrics that poorly capture the semantic precision required for complex QA tasks. These approaches fail to diagnose whether an error stems from faulty retrieval or flawed generation, limiting developers from performing targeted improvement. To address this gap, we propose RAG-X, a diagnostic framework that evaluates the retriever and generator independently across a triad of QA tasks: information extraction, short-answer generation, and multiple-choice question (MCQ) answering. RAG-X introduces Context Utilization Efficiency (CUE) metrics to disaggregate system success into interpretable quadrants, isolating verified grounding from deceptive accuracy. Our experiments reveal an ``Accuracy Fallacy", where a 14\% gap separates perceived system success from evidence-based grounding. By surfacing hidden failure modes, RAG-X offers the diagnostic transparency needed for safe and verifiable clinical RAG systems.