Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection

TL;DR

This paper introduces a social psychology-inspired framework to systematically compare explicit and implicit biases in large language models, revealing significant differences and underlying factors affecting bias manifestation.

Contribution

It presents a novel self-reflection-based evaluation method for implicit bias and uncovers the contrasting behaviors of explicit and implicit biases in LLMs.

Findings

Implicit bias is stronger and more persistent than explicit bias.

Explicit bias decreases with larger training data and models, while implicit bias increases.

Alignment techniques reduce explicit bias but have limited impact on implicit bias.

Abstract

Large Language Models (LLMs) have been shown to exhibit various biases and stereotypes in their generated content. While extensive research has investigated biases in LLMs, prior work has predominantly focused on explicit bias, with minimal attention to implicit bias and the relation between these two forms of bias. This paper presents a systematic framework grounded in social psychology theories to investigate and compare explicit and implicit biases in LLMs. We propose a novel self-reflection-based evaluation framework that operates in two phases: first measuring implicit bias through simulated psychological assessment methods, then evaluating explicit bias by prompting LLMs to analyze their own generated content. Through extensive experiments on advanced LLMs across multiple social dimensions, we demonstrate that LLMs exhibit a substantial inconsistency between explicit and implicit biases: while explicit bias manifests as mild stereotypes, implicit bias exhibits strong stereotypes. We further investigate the underlying factors contributing to this explicit-implicit bias inconsistency, examining the effects of training data scale, model size, and alignment techniques. Experimental results indicate that while explicit bias declines with increased training data and model size, implicit bias exhibits a contrasting upward trend. Moreover, contemporary alignment methods effectively suppress explicit bias but show limited efficacy in mitigating implicit bias.