A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering

TL;DR

This paper systematically evaluates retrieval-augmented medical question answering, analyzing various components and configurations, and demonstrates that effective retrieval strategies significantly enhance zero-shot performance with modest computational resources.

Contribution

It provides a comprehensive analysis of retrieval components in medical QA, identifying optimal configurations and highlighting the tradeoffs between effectiveness and computational cost.

Findings

Dense retrieval with query reformulation and reranking achieved 60.49% accuracy.

Retrieval augmentation significantly improves zero-shot medical question answering.

Simpler dense retrieval configurations offer strong performance with higher throughput.

Abstract

Large language models (LLMs) have demonstrated strong capabilities in medical question answering; however, purely parametric models often suffer from knowledge gaps and limited factual grounding. Retrieval-augmented generation (RAG) addresses this limitation by integrating external knowledge retrieval into the reasoning process. Despite increasing interest in RAG-based medical systems, the impact of individual retrieval components on performance remains insufficiently understood. This study presents a systematic evaluation of retrieval-augmented medical question answering using the MedQA USMLE benchmark and a structured textbook-based knowledge corpus. We analyze the interaction between language models, embedding models, retrieval strategies, query reformulation, and cross-encoder reranking within a unified experimental framework comprising forty configurations. Results show that retrieval augmentation significantly improves zero-shot medical question answering performance. The best-performing configuration was dense retrieval with query reformulation and reranking achieved 60.49% accuracy. Domain-specialized language models were also found to better utilize retrieved medical evidence than general-purpose models. The analysis further reveals a clear tradeoff between retrieval effectiveness and computational cost, with simpler dense retrieval configurations providing strong performance while maintaining higher throughput. All experiments were conducted on a single consumer-grade GPU, demonstrating that systematic evaluation of retrieval-augmented medical QA systems can be performed under modest computational resources.