Where Fact Ends and Fairness Begins: Redefining AI Bias Evaluation through Cognitive Biases

TL;DR

This paper introduces Fact-or-Fair, a benchmark that distinguishes between factual correctness and normative fairness in AI outputs, grounded in cognitive biases, to improve fairness evaluation.

Contribution

It proposes a new benchmark and theoretical framework that separate factual and fairness judgments, addressing limitations of existing fairness assessments.

Findings

Models show varying fact-fair trade-offs

Cognitive biases explain fact-fair misalignments

Benchmark enables nuanced fairness evaluation

Abstract

Recent failures such as Google Gemini generating people of color in Nazi-era uniforms illustrate how AI outputs can be factually plausible yet socially harmful. AI models are increasingly evaluated for "fairness," yet existing benchmarks often conflate two fundamentally different dimensions: factual correctness and normative fairness. A model may generate responses that are factually accurate but socially unfair, or conversely, appear fair while distorting factual reality. We argue that identifying the boundary between fact and fair is essential for meaningful fairness evaluation. We introduce Fact-or-Fair, a benchmark with (i) objective queries aligned with descriptive, fact-based judgments, and (ii) subjective queries aligned with normative, fairness-based judgments. Our queries are constructed from 19 statistics and are grounded in cognitive psychology, drawing on representativeness bias, attribution bias, and ingroup-outgroup bias to explain why models often misalign fact and fairness. Experiments across ten frontier models reveal different levels of fact-fair trade-offs. By reframing fairness evaluation, we provide both a new theoretical lens and a practical benchmark to advance the responsible model assessments. Our test suite is publicly available at https://github.com/uclanlp/Fact-or-Fair.