Do Biased Models Have Biased Thoughts?

TL;DR

This paper investigates whether large language models' biased outputs are reflected in their internal reasoning steps, finding that bias in thinking is not strongly correlated with biased outputs.

Contribution

It introduces an analysis of the relationship between internal reasoning steps and output bias in language models using fairness metrics.

Findings

Bias in reasoning steps is weakly correlated with output bias

Models with biased outputs do not necessarily have biased thoughts

Bias in thinking is not a reliable indicator of output bias

Abstract

The impressive performance of language models is undeniable. However, the presence of biases based on gender, race, socio-economic status, physical appearance, and sexual orientation makes the deployment of language models challenging. This paper studies the effect of chain-of-thought prompting, a recent approach that studies the steps followed by the model before it responds, on fairness. More specifically, we ask the following question: $\textit{Do biased models have biased thoughts}$? To answer our question, we conduct experiments on $5$ popular large language models using fairness metrics to quantify $11$ different biases in the model's thoughts and output. Our results show that the bias in the thinking steps is not highly correlated with the output bias (less than $0.6$ correlation with a $p$-value smaller than $0.001$ in most cases). In other words, unlike human beings, the tested models with biased decisions do not always possess biased thoughts.