Real World Human-LLM Interactions – Prospective blinded versus unblinded expert physician assessments of LLM responses to complex medical dilemmas

TL;DR

This study explores how physicians rate responses from large language models (LLMs) in real clinical scenarios, finding that physician satisfaction does not reliably reflect the quality of LLM-generated medical content.

Contribution

The study introduces a novel approach to evaluating LLMs in healthcare by comparing physician ratings of LLM and human-generated responses in a blinded setting.

Findings

Physician satisfaction scores were similar for LLM and human-generated responses in a blinded evaluation.

Satisfaction did not correlate with the accuracy of cited literature in the responses.

Physician resistance to change did not affect their ratings of LLM responses.

Abstract

Current evaluations of large language models (LLMs) in healthcare have largely emphasized theoretical benchmarks and clinician oversight, with limited exploration of real-world physician-AI interaction. In this two-stage prospective study, we assessed physician satisfaction with LLM-generated responses to real clinical queries. This study did not evaluate clinical accuracy, patient outcomes, or patient safety. In the first unblinded stage, physicians used three models - a general-purpose model (GPT-4o), a reasoning-focused model (GPT-o1), and a healthcare-specific model (OpenEvidence) - to address 25 clinical dilemmas - and rated the quality of the responses. In the second blinded stage, the same physicians evaluated responses generated either by an LLM or by a human alone, without knowledge of the source. Across 100 real-world medical responses, median physician scores on a 5-point Likert scale were comparable between unblinded and blinded evaluations (p = 0.90). Satisfaction was not associated with physicians’ resistance to change, nor did it correlate with the accuracy or relevance of cited literature. These findings suggest that physicians did not favor information generated by LLMs over externally provided responses, and that clinician satisfaction alone may not serve as a reliable proxy for validating decision support tools.

Large language models (LLMs) are increasingly being explored as decision support tools in clinical medicine, yet little is known about how physicians experience them when used for real patient care. In this study, we asked physicians to use three LLMs to help address real clinical dilemmas encountered in their practice, and to rate the quality of the responses. In a second phase, physicians evaluated responses without knowing whether they were generated by an LLM or a human, and their satisfaction scores were similar in both conditions. Notably, satisfaction did not correlate with the accuracy of citations provided in the responses, and physicians’ general resistance to change did not influence their ratings. These findings suggest that physician satisfaction is an insufficient proxy for the actual quality of LLM-generated clinical content. As LLMs move closer to routine clinical use, relying on clinician satisfaction surveys as a validation method may be potentially misleading when assessing whether these tools are appropriate for supporting medical decision-making.

Figures