Gender Trouble in Language Models: An Empirical Audit Guided by Gender Performativity Theory

TL;DR

This paper critically examines how language models encode gender, revealing that they often reinforce binary and biological sex-based stereotypes, which can harm gender diverse identities and calls for broader definitions and mitigation strategies.

Contribution

It introduces an empirical audit guided by gender performativity theory to analyze how models encode gender beyond superficial associations, highlighting the reinforcement of binary and biological sex stereotypes.

Findings

Models tend to encode gender as binary and biologically linked.

Gender non-conforming terms are often erased or pathologized.

Larger models learn stronger gender-sex associations.

Abstract

Language models encode and subsequently perpetuate harmful gendered stereotypes. Research has succeeded in mitigating some of these harms, e.g. by dissociating non-gendered terms such as occupations from gendered terms such as 'woman' and 'man'. This approach, however, remains superficial given that associations are only one form of prejudice through which gendered harms arise. Critical scholarship on gender, such as gender performativity theory, emphasizes how harms often arise from the construction of gender itself, such as conflating gender with biological sex. In language models, these issues could lead to the erasure of transgender and gender diverse identities and cause harms in downstream applications, from misgendering users to misdiagnosing patients based on wrong assumptions about their anatomy. For FAccT research on gendered harms to go beyond superficial linguistic associations, we advocate for a broader definition of 'gender bias' in language models. We operationalize insights on the construction of gender through language from gender studies literature and then empirically test how 16 language models of different architectures, training datasets, and model sizes encode gender. We find that language models tend to encode gender as a binary category tied to biological sex, and that gendered terms that do not neatly fall into one of these binary categories are erased and pathologized. Finally, we show that larger models, which achieve better results on performance benchmarks, learn stronger associations between gender and sex, further reinforcing a narrow understanding of gender. Our findings lead us to call for a re-evaluation of how gendered harms in language models are defined and addressed.