Retrieval-augmented systems can be dangerous medical communicators

TL;DR

Retrieval-augmented AI systems in healthcare can produce accurate yet misleading responses, risking patient misunderstanding despite factual correctness, due to decontextualization and source omission.

Contribution

This paper demonstrates that current retrieval-augmented medical AI systems can mislead patients despite factual accuracy, highlighting the need for improved communication and source comprehension.

Findings

Models decontextualize facts, leading to misinterpretation.

AI responses often omit critical sources, affecting understanding.

Current systems reinforce misconceptions and biases.

Abstract

Patients have long sought health information online, and increasingly, they are turning to generative AI to answer their health-related queries. Given the high stakes of the medical domain, techniques like retrieval-augmented generation and citation grounding have been widely promoted as methods to reduce hallucinations and improve the accuracy of AI-generated responses and have been widely adopted into search engines. This paper argues that even when these methods produce literally accurate content drawn from source documents sans hallucinations, they can still be highly misleading. Patients may derive significantly different interpretations from AI-generated outputs than they would from reading the original source material, let alone consulting a knowledgeable clinician. Through a large-scale query analysis on topics including disputed diagnoses and procedure safety, we support our argument with quantitative and qualitative evidence of the suboptimal answers resulting from current systems. In particular, we highlight how these models tend to decontextualize facts, omit critical relevant sources, and reinforce patient misconceptions or biases. We propose a series of recommendations -- such as the incorporation of communication pragmatics and enhanced comprehension of source documents -- that could help mitigate these issues and extend beyond the medical domain.