SocioEval: A Template-Based Framework for Evaluating Socioeconomic Status Bias in Foundation Models

TL;DR

SocioEval is a comprehensive, template-based framework designed to systematically evaluate socioeconomic bias in foundation models across decision-making tasks, revealing significant bias variation and the effectiveness of safeguards.

Contribution

The paper introduces SocioEval, a novel hierarchical, template-based framework for assessing socioeconomic bias in large language models through structured prompts and evaluation protocols.

Findings

Bias rates vary from 0.42% to 33.75% across models.

Lifestyle judgments exhibit 10× higher bias than education decisions.

Safeguards prevent explicit discrimination but are brittle to stereotypes.

Abstract

As Large Language Models (LLMs) increasingly power decision-making systems across critical domains, understanding and mitigating their biases becomes essential for responsible AI deployment. Although bias assessment frameworks have proliferated for attributes such as race and gender, socioeconomic status bias remains significantly underexplored despite its widespread implications in the real world. We introduce SocioEval, a template-based framework for systematically evaluating socioeconomic bias in foundation models through decision-making tasks. Our hierarchical framework encompasses 8 themes and 18 topics, generating 240 prompts across 6 class-pair combinations. We evaluated 13 frontier LLMs on 3,120 responses using a rigorous three-stage annotation protocol, revealing substantial variation in bias rates (0.42\%-33.75\%). Our findings demonstrate that bias manifests differently across themes lifestyle judgments show 10$\times$ higher bias than education-related decisions and that deployment safeguards effectively prevent explicit discrimination but show brittleness to domain-specific stereotypes. SocioEval provides a scalable, extensible foundation for auditing class-based bias in language models.