Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence

TL;DR

This paper examines how large language models respond to counterfactual medical evidence, revealing they often accept false or dangerous information at face value, which raises safety concerns.

Contribution

The authors introduce MedCounterFact, a novel dataset for evaluating LLMs' reasoning with counterfactual medical evidence, highlighting models' overreliance on provided context.

Findings

Models accept counterfactual evidence even when dangerous or implausible.

LLMs provide confident answers without caveats in the presence of false evidence.

Existing models overemphasize faithfulness over safety in high-stakes contexts.

Abstract

In high-stakes domains like medicine, it may be generally desirable for models to faithfully adhere to the context provided. But what happens if the context does not align with model priors or safety protocols? In this paper, we investigate how LLMs behave and reason when presented with counterfactual (or even adversarial) medical evidence. We first construct MedCounterFact, a counterfactual medical QA dataset that requires the models to answer clinical comparison questions (i.e., judge the efficacy of certain treatments, with evidence consisting of randomized controlled trials provided as context). In MedCounterFact, real-world medical interventions within the questions and evidence are systematically replaced with four types of counterfactual stimuli, ranging from unknown words to toxic substances. Our evaluation across multiple frontier LLMs on MedCounterFact reveals that in the presence of counterfactual evidence, existing models overwhelmingly accept such "evidence" at face value even when it is dangerous or implausible, and provide confident and uncaveated answers. While it may be prudent to draw a boundary between faithfulness and safety, our findings suggest that models arguably overemphasize the former.