Rationale-Guided Retrieval Augmented Generation for Medical Question Answering

TL;DR

RAG$^2$ enhances biomedical question answering by integrating rationale-guided retrieval, filtering, and balanced corpus sampling, significantly improving accuracy over existing models.

Contribution

Introduces RAG$^2$, a novel framework that uses rationale-guided filtering and balanced retrieval to improve medical question answering accuracy.

Findings

RAG$^2$ outperforms previous models by up to 5.6% on medical QA benchmarks.

Increases accuracy of LLMs by up to 6.1%.

Effectively mitigates retriever bias across multiple biomedical corpora.

Abstract

Large language models (LLM) hold significant potential for applications in biomedicine, but they struggle with hallucinations and outdated knowledge. While retrieval-augmented generation (RAG) is generally employed to address these issues, it also has its own set of challenges: (1) LLMs are vulnerable to irrelevant or incorrect context, (2) medical queries are often not well-targeted for helpful information, and (3) retrievers are prone to bias toward the specific source corpus they were trained on. In this study, we present RAG$^2$ (RAtionale-Guided RAG), a new framework for enhancing the reliability of RAG in biomedical contexts. RAG$^2$ incorporates three key innovations: a small filtering model trained on perplexity-based labels of rationales, which selectively augments informative snippets of documents while filtering out distractors; LLM-generated rationales as queries to improve the utility of retrieved snippets; a structure designed to retrieve snippets evenly from a comprehensive set of four biomedical corpora, effectively mitigating retriever bias. Our experiments demonstrate that RAG$^2$ improves the state-of-the-art LLMs of varying sizes, with improvements of up to 6.1\%, and it outperforms the previous best medical RAG model by up to 5.6\% across three medical question-answering benchmarks. Our code is available at https://github.com/dmis-lab/RAG2.