Efficient and Reproducible Biomedical Question Answering using Retrieval Augmented Generation

TL;DR

This paper evaluates retrieval-augmented generation methods for biomedical question answering, optimizing retrieval strategies and response times on large PubMed datasets to improve accuracy, efficiency, and scalability.

Contribution

It systematically compares retrieval strategies and demonstrates an optimal balance between accuracy and response time in biomedical QA using RAG systems.

Findings

Retrieving 50 documents with BM25 and reranking with MedCPT balances accuracy and speed.

BM25 retrieval remains fast at 82ms, while MedCPT adds computational cost.

The study provides insights into retrieval depth, efficiency, and scalability trade-offs.

Abstract

Biomedical question-answering (QA) systems require effective retrieval and generation components to ensure accuracy, efficiency, and scalability. This study systematically examines a Retrieval-Augmented Generation (RAG) system for biomedical QA, evaluating retrieval strategies and response time trade-offs. We first assess state-of-the-art retrieval methods, including BM25, BioBERT, MedCPT, and a hybrid approach, alongside common data stores such as Elasticsearch, MongoDB, and FAISS, on a ~10% subset of PubMed (2.4M documents) to measure indexing efficiency, retrieval latency, and retriever performance in the end-to-end RAG system. Based on these insights, we deploy the final RAG system on the full 24M PubMed corpus, comparing different retrievers' impact on overall performance. Evaluations of the retrieval depth show that retrieving 50 documents with BM25 before reranking with MedCPT optimally balances accuracy (0.90), recall (0.90), and response time (1.91s). BM25 retrieval time remains stable (82ms), while MedCPT incurs the main computational cost. These results highlight previously not well-known trade-offs in retrieval depth, efficiency, and scalability for biomedical QA. With open-source code, the system is fully reproducible and extensible.