Veracity Bias and Beyond: Uncovering LLMs' Hidden Beliefs in Problem-Solving Reasoning

TL;DR

This paper uncovers hidden biases in large language models, showing they associate solution correctness with demographics, which could impact their use in educational and evaluative contexts.

Contribution

It identifies two forms of veracity bias in LLMs—Attribution and Evaluation Bias—and demonstrates these biases are deeply embedded in their reasoning processes.

Findings

LLMs attribute fewer correct solutions to African-American groups in math and coding.

LLMs prefer Asian authorship in writing evaluation.

Models assign racially stereotypical colors in visualization code.

Abstract

Despite LLMs' explicit alignment against demographic stereotypes, they have been shown to exhibit biases under various social contexts. In this work, we find that LLMs exhibit concerning biases in how they associate solution veracity with demographics. Through experiments across five human value-aligned LLMs on mathematics, coding, commonsense, and writing problems, we reveal two forms of such veracity biases: Attribution Bias, where models disproportionately attribute correct solutions to certain demographic groups, and Evaluation Bias, where models' assessment of identical solutions varies based on perceived demographic authorship. Our results show pervasive biases: LLMs consistently attribute fewer correct solutions and more incorrect ones to African-American groups in math and coding, while Asian authorships are least preferred in writing evaluation. In additional studies, we show LLMs automatically assign racially stereotypical colors to demographic groups in visualization code, suggesting these biases are deeply embedded in models' reasoning processes. Our findings indicate that demographic bias extends beyond surface-level stereotypes and social context provocations, raising concerns about LLMs' deployment in educational and evaluation settings.