Contextual StereoSet: Stress-Testing Bias Alignment Robustness in Large Language Models

TL;DR

This paper introduces Contextual StereoSet, a benchmark for stress-testing bias robustness in large language models by varying contextual framing, revealing significant bias shifts and proposing a diagnostic profile for evaluation consistency.

Contribution

The paper presents a novel benchmark and methodology for evaluating bias robustness in language models under different contextual conditions, highlighting the variability of bias detection.

Findings

Bias shifts significantly with contextual framing.

Certain frames increase stereotype selection across models.

Evaluation scores may not generalize across different contexts.

Abstract

A model that avoids stereotypes in a lab benchmark may not avoid them in deployment. We show that measured bias shifts dramatically when prompts mention different places, times, or audiences -- no adversarial prompting required. We introduce Contextual StereoSet, a benchmark that holds stereotype content fixed while systematically varying contextual framing. Testing 13 models across two protocols, we find striking patterns: anchoring to 1990 (vs. 2030) raises stereotype selection in all models tested on this contrast (p<0.05); gossip framing raises it in 5 of 6 full-grid models; out-group observer framing shifts it by up to 13 percentage points. These effects replicate in hiring, lending, and help-seeking vignettes. We propose Context Sensitivity Fingerprints (CSF): a compact profile of per-dimension dispersion and paired contrasts with bootstrap CIs and FDR correction. Two evaluation tracks support different use cases -- a 360-context diagnostic grid for deep analysis and a budgeted protocol covering 4,229 items for production screening. The implication is methodological: bias scores from fixed-condition tests may not generalize.This is not a claim about ground-truth bias rates; it is a stress test of evaluation robustness. CSF forces evaluators to ask, "Under what conditions does bias appear?" rather than "Is this model biased?" We release our benchmark, code, and results.