Evaluating Implicit Biases in LLM Reasoning through Logic Grid Puzzles

TL;DR

This paper introduces PRIME, a new framework using logic grid puzzles to systematically evaluate and quantify implicit social biases, especially gender stereotypes, in large language models' reasoning processes.

Contribution

We propose PRIME, an innovative evaluation method that leverages logic puzzles to detect subtle social biases in LLM reasoning, enabling controlled, automated, and nuanced bias analysis.

Findings

Models reason more accurately with stereotypical solutions.

PRIME effectively reveals gender biases in LLM reasoning.

The framework allows for controlled bias comparisons.

Abstract

While recent safety guardrails effectively suppress overtly biased outputs, subtler forms of social bias emerge during complex logical reasoning tasks that evade current evaluation benchmarks. To fill this gap, we introduce a new evaluation framework, PRIME (Puzzle Reasoning for Implicit Biases in Model Evaluation), that uses logic grid puzzles to systematically probe the influence of social stereotypes on logical reasoning and decision making in LLMs. Our use of logic puzzles enables automatic generation and verification, as well as variability in complexity and biased settings. PRIME includes stereotypical, anti-stereotypical, and neutral puzzle variants generated from a shared puzzle structure, allowing for controlled and fine-grained comparisons. We evaluate multiple model families across puzzle sizes and test the effectiveness of prompt-based mitigation strategies. Focusing our experiments on gender stereotypes, our findings highlight that models consistently reason more accurately when solutions align with stereotypical associations. This demonstrates the significance of PRIME for diagnosing and quantifying social biases perpetuated in the deductive reasoning of LLMs, where fairness is critical.