Question answering systems for health professionals at the point of care -- a systematic review

TL;DR

This systematic review evaluates current medical question answering systems for healthcare, highlighting their limitations in real-world applicability, reliability, and clinician involvement, and suggests directions for future research to improve clinical utility.

Contribution

The paper provides a comprehensive analysis of existing biomedical QA systems, identifying gaps in clinical relevance, evaluation methods, and user involvement, and emphasizes the need for more realistic datasets and source reliability.

Findings

Most systems lack communication of confidence levels.

High risk of bias and limited clinical utility in many studies.

Few systems involve clinicians in development and evaluation.

Abstract

Objective: Question answering (QA) systems have the potential to improve the quality of clinical care by providing health professionals with the latest and most relevant evidence. However, QA systems have not been widely adopted. This systematic review aims to characterize current medical QA systems, assess their suitability for healthcare, and identify areas of improvement. Materials and methods: We searched PubMed, IEEE Xplore, ACM Digital Library, ACL Anthology and forward and backward citations on 7th February 2023. We included peer-reviewed journal and conference papers describing the design and evaluation of biomedical QA systems. Two reviewers screened titles, abstracts, and full-text articles. We conducted a narrative synthesis and risk of bias assessment for each study. We assessed the utility of biomedical QA systems. Results: We included 79 studies and identified themes, including question realism, answer reliability, answer utility, clinical specialism, systems, usability, and evaluation methods. Clinicians' questions used to train and evaluate QA systems were restricted to certain sources, types and complexity levels. No system communicated confidence levels in the answers or sources. Many studies suffered from high risks of bias and applicability concerns. Only 8 studies completely satisfied any criterion for clinical utility, and only 7 reported user evaluations. Most systems were built with limited input from clinicians. Discussion: While machine learning methods have led to increased accuracy, most studies imperfectly reflected real-world healthcare information needs. Key research priorities include developing more realistic healthcare QA datasets and considering the reliability of answer sources, rather than merely focusing on accuracy.