The Lou Dataset -- Exploring the Impact of Gender-Fair Language in German Text Classification

TL;DR

This paper introduces Lou, a new dataset of gender-fair language reformulations in German text classification, revealing significant impacts on model predictions and attention patterns, with implications for multilingual NLP.

Contribution

The paper presents Lou, the first dataset of high-quality German gender-fair language reformulations for seven classification tasks, enabling assessment of linguistic shifts on language models.

Findings

Gender-fair language significantly affects model predictions and attention.

Existing model rankings remain stable despite linguistic shifts.

Findings likely extend to other languages and multilingual models.

Abstract

Gender-fair language, an evolving German linguistic variation, fosters inclusion by addressing all genders or using neutral forms. Nevertheless, there is a significant lack of resources to assess the impact of this linguistic shift on classification using language models (LMs), which are probably not trained on such variations. To address this gap, we present Lou, the first dataset featuring high-quality reformulations for German text classification covering seven tasks, like stance detection and toxicity classification. Evaluating 16 mono- and multi-lingual LMs on Lou shows that gender-fair language substantially impacts predictions by flipping labels, reducing certainty, and altering attention patterns. However, existing evaluations remain valid, as LM rankings of original and reformulated instances do not significantly differ. While we offer initial insights on the effect on German text classification, the findings likely apply to other languages, as consistent patterns were observed in multi-lingual and English LMs.