How well do LLMs cite relevant medical references? An evaluation framework and analyses

TL;DR

This paper evaluates how accurately large language models cite relevant medical references, developing an automated pipeline and expert validation to assess the supportiveness of generated sources, revealing significant gaps in source reliability.

Contribution

It introduces SourceCheckup, an automated evaluation pipeline, and demonstrates that many LLM responses are not fully supported by their cited sources in medical contexts.

Findings

88% agreement between GPT-4 and medical experts on source relevance

50-90% of LLM responses lack full source support

Approximately 30% of GPT-4 statements are unsupported even with retrieval augmentation

Abstract

Large language models (LLMs) are currently being used to answer medical questions across a variety of clinical domains. Recent top-performing commercial LLMs, in particular, are also capable of citing sources to support their responses. In this paper, we ask: do the sources that LLMs generate actually support the claims that they make? To answer this, we propose three contributions. First, as expert medical annotations are an expensive and time-consuming bottleneck for scalable evaluation, we demonstrate that GPT-4 is highly accurate in validating source relevance, agreeing 88% of the time with a panel of medical doctors. Second, we develop an end-to-end, automated pipeline called \textit{SourceCheckup} and use it to evaluate five top-performing LLMs on a dataset of 1200 generated questions, totaling over 40K pairs of statements and sources. Interestingly, we find that between ~50% to 90% of LLM responses are not fully supported by the sources they provide. We also evaluate GPT-4 with retrieval augmented generation (RAG) and find that, even still, around 30\% of individual statements are unsupported, while nearly half of its responses are not fully supported. Third, we open-source our curated dataset of medical questions and expert annotations for future evaluations. Given the rapid pace of LLM development and the potential harms of incorrect or outdated medical information, it is crucial to also understand and quantify their capability to produce relevant, trustworthy medical references.