Race, Ethnicity and Their Implication on Bias in Large Language Models

TL;DR

This study investigates how large language models encode race and ethnicity internally, revealing diverse representations and the partial effectiveness of interventions in reducing bias, highlighting the need for systematic mitigation strategies.

Contribution

It provides a mechanistic analysis of demographic representation in LLMs using interpretability techniques, revealing internal encoding and intervention effects on bias.

Findings

Demographic info is distributed across internal units with variation.

Some units encode stereotypes from pretraining.

Interventions reduce bias but leave residual effects.

Abstract

Large language models (LLMs) increasingly operate in high-stakes settings including healthcare and medicine, where demographic attributes such as race and ethnicity may be explicitly stated or implicitly inferred from text. However, existing studies primarily document outcome-level disparities, offering limited insight into internal mechanisms underlying these effects. We present a mechanistic study of how race and ethnicity are represented and operationalized within LLMs. Using two publicly available datasets spanning toxicity-related generation and clinical narrative understanding tasks, we analyze three open-source models with a reproducible interpretability pipeline combining probing, neuron-level attribution, and targeted intervention. We find that demographic information is distributed across internal units with substantial cross-model variation. Although some units encode sensitive or stereotype-related associations from pretraining, identical demographic cues can induce qualitatively different behaviors. Interventions suppressing such neurons reduce bias but leave substantial residual effects, suggesting behavioral rather than representational change and motivating more systematic mitigation.