A Counterfactual LLM Framework for Detecting Human Biases: A Case Study of Sex/Gender in Emergency Triage

TL;DR

This paper introduces a counterfactual LLM-based framework to detect gender biases in clinical decision-making, validated on hospital data and applicable across various domains to identify and address disparities.

Contribution

It presents a novel, domain-agnostic counterfactual method using LLMs to quantify gender disparities in decision-making processes.

Findings

Women are approximately 2.1% less likely to receive higher-severity triage scores.

The approach detects both explicit and implicit gender cues contributing to disparities.

Over 200,000 lower-severity triage assignments per year in France are linked to gender bias.

Abstract

We present a novel, domain-agnostic counterfactual approach that uses Large Language Models (LLMs) to quantify gender disparities in human clinical decision-making. The method trains an LLM to emulate observed decisions, then evaluates counterfactual pairs in which only gender is flipped, estimating directional disparities while holding all other clinical factors constant. We study emergency triage, validating the approach on more than 150,000 admissions to the Bordeaux University Hospital (France) and replicating results on a subset of MIMIC-IV across a different language, population, and healthcare system. In the Bordeaux cohort, otherwise identical presentations were approximately 2.1% more likely to receive a lower-severity triage score when presented as female rather than male; scaled to national emergency volumes in France, this corresponds to more than 200,000 lower-severity assignments per year. Modality-specific analyses indicate that both explicit tabular gender indicators and implicit textual gender cues contribute to the disparity. Beyond emergency care, the approach supports bias audits in other settings (e.g., hiring, academic, and justice decisions), providing a scalable tool to detect and address inequities in real-world decision-making.