The Gender-GAP Pipeline: A Gender-Aware Polyglot Pipeline for Gender Characterisation in 55 Languages

TL;DR

The paper introduces the Gender-GAP Pipeline, an automatic tool for quantifying gender representation in large multilingual datasets, revealing biases towards masculine terms and advocating for balanced data to mitigate gender bias in language systems.

Contribution

It presents a novel multilingual pipeline for measuring gender representation in large datasets across 55 languages, aiding bias detection and mitigation.

Findings

Current datasets are skewed towards masculine gender representation.

The pipeline effectively quantifies gender bias in large-scale multilingual data.

Using the pipeline can inform data balancing efforts to reduce gender bias.

Abstract

Gender biases in language generation systems are challenging to mitigate. One possible source for these biases is gender representation disparities in the training and evaluation data. Despite recent progress in documenting this problem and many attempts at mitigating it, we still lack shared methodology and tooling to report gender representation in large datasets. Such quantitative reporting will enable further mitigation, e.g., via data augmentation. This paper describes the Gender-GAP Pipeline (for Gender-Aware Polyglot Pipeline), an automatic pipeline to characterize gender representation in large-scale datasets for 55 languages. The pipeline uses a multilingual lexicon of gendered person-nouns to quantify the gender representation in text. We showcase it to report gender representation in WMT training data and development data for the News task, confirming that current data is skewed towards masculine representation. Having unbalanced datasets may indirectly optimize our systems towards outperforming one gender over the others. We suggest introducing our gender quantification pipeline in current datasets and, ideally, modifying them toward a balanced representation.