Explanatory Argument Extraction of Correct Answers in Resident Medical Exams

TL;DR

This paper introduces a new Spanish medical exam dataset with explanations for correct and incorrect answers, enabling automated evaluation of language models' ability to extract relevant medical explanations, aiding practitioners.

Contribution

The work presents a novel dataset with doctor-written explanations for both correct and incorrect answers in Spanish medical exams, facilitating extractive QA tasks and automated model evaluation.

Findings

Multilingual models sometimes outperform monolingual models.

Smaller models can perform competitively despite being inferior.

The dataset effectively helps identify relevant evidence-based medical explanations.

Abstract

Developing the required technology to assist medical experts in their everyday activities is currently a hot topic in the Artificial Intelligence research field. Thus, a number of large language models (LLMs) and automated benchmarks have recently been proposed with the aim of facilitating information extraction in Evidence-Based Medicine (EBM) using natural language as a tool for mediating in human-AI interaction. The most representative benchmarks are limited to either multiple-choice or long-form answers and are available only in English. In order to address these shortcomings, in this paper we present a new dataset which, unlike previous work: (i) includes not only explanatory arguments for the correct answer, but also arguments to reason why the incorrect answers are not correct; (ii) the explanations are written originally by medical doctors to answer questions from the Spanish Residency Medical Exams. Furthermore, this new benchmark allows us to setup a novel extractive task which consists of identifying the explanation of the correct answer written by medical doctors. An additional benefit of our setting is that we can leverage the extractive QA paradigm to automatically evaluate performance of LLMs without resorting to costly manual evaluation by medical experts. Comprehensive experimentation with language models for Spanish shows that sometimes multilingual models fare better than monolingual ones, even outperforming models which have been adapted to the medical domain. Furthermore, results across the monolingual models are mixed, with supposedly smaller and inferior models performing competitively. In any case, the obtained results show that our novel dataset and approach can be an effective technique to help medical practitioners in identifying relevant evidence-based explanations for medical questions.