Measuring Gender Bias in Word Embeddings of Gendered Languages Requires Disentangling Grammatical Gender Signals

TL;DR

This paper investigates how grammatical gender influences the measurement of social gender bias in word embeddings of gendered languages, proposing methods to disentangle these signals for more accurate bias assessment.

Contribution

It introduces post-processing techniques to separate grammatical gender signals from semantic gender information in word embeddings of multiple languages.

Findings

Disentangling grammatical gender reduces bias effect size significantly.

Over 90% of inanimate nouns show weakened gender associations after disentangling.

Cross-lingual bias results align better with country-level implicit bias measurements.

Abstract

Does the grammatical gender of a language interfere when measuring the semantic gender information captured by its word embeddings? A number of anomalous gender bias measurements in the embeddings of gendered languages suggest this possibility. We demonstrate that word embeddings learn the association between a noun and its grammatical gender in grammatically gendered languages, which can skew social gender bias measurements. Consequently, word embedding post-processing methods are introduced to quantify, disentangle, and evaluate grammatical gender signals. The evaluation is performed on five gendered languages from the Germanic, Romance, and Slavic branches of the Indo-European language family. Our method reduces the strength of grammatical gender signals, which is measured in terms of effect size (Cohen's d), by a significant average of d = 1.3 for French, German, and Italian, and d = 0.56 for Polish and Spanish. Once grammatical gender is disentangled, the association between over 90% of 10,000 inanimate nouns and their assigned grammatical gender weakens, and cross-lingual bias results from the Word Embedding Association Test (WEAT) become more congruent with country-level implicit bias measurements. The results further suggest that disentangling grammatical gender signals from word embeddings may lead to improvement in semantic machine learning tasks.