Failure of contextual invariance in gender inference with large language models

TL;DR

This paper demonstrates that large language models' gender inference outputs are highly sensitive to subtle contextual changes, challenging assumptions of output stability and raising concerns for bias evaluation and real-world deployment.

Contribution

The study reveals that LLMs violate contextual invariance in gender inference tasks, showing systematic output shifts due to minimal context variations, which was previously unrecognized.

Findings

Large, systematic shifts in model outputs caused by minimal context changes.

Weakening or disappearance of correlations with cultural stereotypes in contextual settings.

Dependence on irrelevant features like pronoun gender persists in many cases.

Abstract

Standard evaluation practices assume that large language model (LLM) outputs are stable under contextually equivalent formulations of a task. Here, we test this assumption in the setting of gender inference. Using a controlled pronoun selection task, we introduce minimal, theoretically uninformative discourse context and find that this induces large, systematic shifts in model outputs. Correlations with cultural gender stereotypes, present in decontextualized settings, weaken or disappear once context is introduced, while theoretically irrelevant features, such as the gender of a pronoun for an unrelated referent, become the most informative predictors of model behaviour. A Contextuality-by-Default analysis reveals that, in 19--52\% of cases across models, this dependence persists after accounting for all marginal effects of context on individual outputs and cannot be attributed to simple pronoun repetition. These findings show that LLM outputs violate contextual invariance even under near-identical syntactic formulations, with implications for bias benchmarking and deployment in high-stakes settings.