Do They Understand Them? An Updated Evaluation on Nonbinary Pronoun Handling in Large Language Models

TL;DR

This paper evaluates how well large language models handle nonbinary and neopronouns, showing progress but also highlighting ongoing challenges in gender-inclusive language understanding.

Contribution

Introduces MISGENDERED+ benchmark, updates evaluation of LLMs' pronoun handling, and compares five models across various tasks, revealing both improvements and persistent gaps.

Findings

Notable improvements in binary and gender-neutral pronoun accuracy.

Inconsistent performance on neopronouns and reverse inference tasks.

Persistent gaps in identity-sensitive reasoning in LLMs.

Abstract

Large language models (LLMs) are increasingly deployed in sensitive contexts where fairness and inclusivity are critical. Pronoun usage, especially concerning gender-neutral and neopronouns, remains a key challenge for responsible AI. Prior work, such as the MISGENDERED benchmark, revealed significant limitations in earlier LLMs' handling of inclusive pronouns, but was constrained to outdated models and limited evaluations. In this study, we introduce MISGENDERED+, an extended and updated benchmark for evaluating LLMs' pronoun fidelity. We benchmark five representative LLMs, GPT-4o, Claude 4, DeepSeek-V3, Qwen Turbo, and Qwen2.5, across zero-shot, few-shot, and gender identity inference. Our results show notable improvements compared with previous studies, especially in binary and gender-neutral pronoun accuracy. However, accuracy on neopronouns and reverse inference tasks remains inconsistent, underscoring persistent gaps in identity-sensitive reasoning. We discuss implications, model-specific observations, and avenues for future inclusive AI research.