Beyond Retrieval: Ensembling Cross-Encoders and GPT Rerankers with LLMs for Biomedical QA

TL;DR

This paper introduces a retrieval-augmented generation system for biomedical question answering that combines dense retrieval, ensemble re-ranking with cross-encoders and LLMs, and few-shot prompting, achieving competitive results in the BioASQ 2025 challenge.

Contribution

It presents a novel ensemble approach for re-ranking biomedical documents and demonstrates effective answer generation using instruction-tuned LLMs in a biomedical QA setting.

Findings

Achieved MAP@10 of 0.1581, ranking 10th in retrieval.

Attained macro-F1 of 0.95 for yes/no questions.

Secured rank 1 in factoid question MRR.

Abstract

Biomedical semantic question answering rooted in information retrieval can play a crucial role in keeping up to date with vast, rapidly evolving and ever-growing biomedical literature. A robust system can help researchers, healthcare professionals and even layman users access relevant knowledge grounded in evidence. The BioASQ 2025 Task13b Challenge serves as an important benchmark, offering a competitive platform for advancement of this space. This paper presents the methodologies and results from our participation in this challenge where we built a Retrieval-Augmented Generation (RAG) system that can answer biomedical questions by retrieving relevant PubMed documents and snippets to generate answers. For the retrieval task, we generated dense embeddings from biomedical articles for initial retrieval, and applied an ensemble of finetuned cross-encoders and large language models (LLMs) for re-ranking to identify top relevant documents. Our solution achieved an MAP@10 of 0.1581, placing 10th on the leaderboard for the retrieval task. For answer generation, we employed few-shot prompting of instruction-tuned LLMs. Our system achieved macro-F1 score of 0.95 for yes/no questions (rank 12), Mean Reciprocal Rank (MRR) of 0.64 for factoid questions (rank 1), mean-F1 score of 0.63 for list questions (rank 5), and ROUGE-SU4 F1 score of 0.29 for ideal answers (rank 11).