What Would it Take to get Biomedical QA Systems into Practice?

TL;DR

This paper discusses the challenges and necessary criteria for making biomedical question answering systems practical and trustworthy for clinical use, emphasizing transparency and reliability.

Contribution

It proposes criteria for practical biomedical QA systems, assesses current models against these, and highlights areas needing improvement for clinical adoption.

Findings

Current models lack transparency and trustworthiness.

Existing datasets and tasks do not fully meet practical criteria.

Addressing these gaps could improve clinical utility.

Abstract

Medical question answering (QA) systems have the potential to answer clinicians uncertainties about treatment and diagnosis on demand, informed by the latest evidence. However, despite the significant progress in general QA made by the NLP community, medical QA systems are still not widely used in clinical environments. One likely reason for this is that clinicians may not readily trust QA system outputs, in part because transparency, trustworthiness, and provenance have not been key considerations in the design of such models. In this paper we discuss a set of criteria that, if met, we argue would likely increase the utility of biomedical QA systems, which may in turn lead to adoption of such systems in practice. We assess existing models, tasks, and datasets with respect to these criteria, highlighting shortcomings of previously proposed approaches and pointing toward what might be more usable QA systems.