MRAG: Benchmarking Retrieval-Augmented Generation for Bio-medicine

TL;DR

The paper introduces MRAG, a comprehensive benchmark and toolkit for evaluating Retrieval-Augmented Generation in biomedical NLP, highlighting its impact on model reliability, retrieval methods, and response quality.

Contribution

It presents the first extensive MRAG benchmark and toolkit for biomedical RAG, covering multiple languages and tasks, and analyzes factors affecting RAG performance.

Findings

RAG improves LLM reliability in biomedical tasks.

Retrieval approach, model size, and prompting influence RAG performance.

RAG enhances usefulness and reasoning but may reduce response readability for long questions.

Abstract

While Retrieval-Augmented Generation (RAG) has been swiftly adopted in scientific and clinical QA systems, a comprehensive evaluation benchmark in the medical domain is lacking. To address this gap, we introduce the Medical Retrieval-Augmented Generation (MRAG) benchmark, covering various tasks in English and Chinese languages, and building a corpus with Wikipedia and Pubmed. Additionally, we develop the MRAG-Toolkit, facilitating systematic exploration of different RAG components. Our experiments reveal that: (a) RAG enhances LLM reliability across MRAG tasks. (b) the performance of RAG systems is influenced by retrieval approaches, model sizes, and prompting strategies. (c) While RAG improves usefulness and reasoning quality, LLM responses may become slightly less readable for long-form questions. We will release the MRAG-Bench's dataset and toolkit with CCBY-4.0 license upon acceptance, to facilitate applications from both academia and industry.