Mechanics of Bias and Reasoning: Interpreting the Impact of Chain-of-Thought Prompting on Gender Bias in LLMs

TL;DR

This paper investigates how chain-of-thought prompting influences gender bias in large language models, revealing that it offers superficial mitigation rather than genuine bias reduction.

Contribution

It combines benchmark evaluation with mechanistic interpretability to analyze the actual impact of CoT prompting on gender bias in LLMs.

Findings

CoT prompting does not consistently reduce gender bias gaps.

Bias remains embedded in hidden representations despite superficial improvements.

Improvements are due to memorization, not genuine understanding.

Abstract

Large language models (LLMs) are increasingly deployed in socially sensitive settings despite substantial documentation that they encode gender biases. Chain-of-Thought (CoT) prompting has been proposed as a bias-mitigation approach. However, existing evaluations primarily focus on changes in LLM benchmark performance, providing limited insight into whether apparent bias reductions reflect meaningful changes in a model's internal mechanisms. In this work, we investigate how CoT prompting affects gender bias in LLMs, combining benchmark-based evaluation with mechanistic interpretability techniques and reasoning chain failure analysis. Our results confirm a stereotypical bias present in LLM outputs across benchmarks, showing that CoT prompting does not consistently reduce the bias gap. Mechanistic analyses reveal that although CoT balances biased behavior in certain attention head clusters, gender bias remains embedded in hidden representations, indicating only superficial mitigation. Inspection of reasoning chains further suggests that these improvements stem from memorization and familiarity with the dataset rather than genuine understanding of bias.