How Deep Is Representational Bias in LLMs? The Cases of Caste and Religion

TL;DR

This study systematically audits GPT-4 Turbo to measure the depth and persistence of representational biases related to caste and religion in India, revealing overrepresentation of dominant groups despite diversity prompts.

Contribution

It provides a novel, large-scale analysis of caste and religious biases in LLMs, extending bias research beyond common identities like race and gender.

Findings

GPT-4 overrepresents dominant groups beyond their population share

Repeated prompts have limited effect on reducing biases

Biases are more entrenched than training data distribution suggests

Abstract

Representational bias in large language models (LLMs) has predominantly been measured through single-response interactions and has focused on Global North-centric identities like race and gender. We expand on that research by conducting a systematic audit of GPT-4 Turbo to reveal how deeply encoded representational biases are and how they extend to less-explored dimensions of identity. We prompt GPT-4 Turbo to generate over 7,200 stories about significant life events (such as weddings) in India, using prompts designed to encourage diversity to varying extents. Comparing the diversity of religious and caste representation in the outputs against the actual population distribution in India as recorded in census data, we quantify the presence and "stickiness" of representational bias in the LLM for religion and caste. We find that GPT-4 responses consistently overrepresent culturally dominant groups far beyond their statistical representation, despite prompts intended to encourage representational diversity. Our findings also suggest that representational bias in LLMs has a winner-take-all quality that is more biased than the likely distribution bias in their training data, and repeated prompt-based nudges have limited and inconsistent efficacy in dislodging these biases. These results suggest that diversifying training data alone may not be sufficient to correct LLM bias, highlighting the need for more fundamental changes in model development. Dataset and Codebook: https://github.com/agrimaseth/How-Deep-Is-Representational-Bias-in-LLMs