Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA

TL;DR

This study reveals that social identity markers like sexual orientation and religious affiliation significantly impair the accuracy and confidence calibration of large language models in medical question answering, risking unsafe clinical deployment.

Contribution

It uncovers how social descriptors distort LLM calibration and accuracy, highlighting risks in equitable healthcare AI deployment.

Findings

Identity markers cause performance drops in LLMs.

Intersectional identities produce non-additive calibration harms.

Failures persist in open-ended generation settings.

Abstract

Safe clinical deployment of Large Language Models (LLMs) requires not only high accuracy but also robust uncertainty calibration to ensure models defer to clinicians when appropriate. Our paper investigates how social descriptors of a patient (specifically sexual orientation and religious affiliation) distort these uncertainty signals and model accuracy. Evaluating nine general-purpose and biomedical LLMs on 2,364 medical questions and their counterfactual variants, we demonstrate that identity markers cause a "calibration crisis". "Homosexual" markers consistently trigger performance drops, and intersectional identities produce idiosyncratic, non-additive harms to calibration. Moreover, a clinician-validated case study in an open-ended generation setting confirms that these failures are not an artifact of the multiple-choice format. Our results demonstrate that the presence of social identity cues does not merely shift predictions; it affects the reliability of confidence signals, posing a significant risk to equitable care and safe deployment in confidence-based clinical workflows.