RealMedQA: A pilot biomedical question answering dataset containing realistic clinical questions

TL;DR

RealMedQA is a new dataset of realistic clinical questions created by humans and an LLM, designed to improve biomedical question answering systems by reflecting real-world clinical needs and challenging existing models.

Contribution

This work introduces RealMedQA, a novel dataset of realistic clinical questions, and evaluates QA models, highlighting the cost-efficiency of LLMs and the increased difficulty of matching answers.

Findings

LLMs are more cost-effective for generating QA pairs.

RealMedQA has lower lexical similarity between questions and answers than BioASQ.

QA models perform less well on RealMedQA, indicating increased challenge.

Abstract

Clinical question answering systems have the potential to provide clinicians with relevant and timely answers to their questions. Nonetheless, despite the advances that have been made, adoption of these systems in clinical settings has been slow. One issue is a lack of question-answering datasets which reflect the real-world needs of health professionals. In this work, we present RealMedQA, a dataset of realistic clinical questions generated by humans and an LLM. We describe the process for generating and verifying the QA pairs and assess several QA models on BioASQ and RealMedQA to assess the relative difficulty of matching answers to questions. We show that the LLM is more cost-efficient for generating "ideal" QA pairs. Additionally, we achieve a lower lexical similarity between questions and answers than BioASQ which provides an additional challenge to the top two QA models, as per the results. We release our code and our dataset publicly to encourage further research.