Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translation

TL;DR

This paper introduces Minimal Pair Accuracy (MPA), a new metric to evaluate how well neural machine translation models incorporate gender cues, revealing biases and attention patterns related to gender disambiguation.

Contribution

The paper proposes MPA as a novel evaluation metric for gender disambiguation in NMT, and analyzes model biases and attention mechanisms regarding gender cues in translation.

Findings

Models often ignore gender cues in favor of stereotypes.

Anti-stereotypical cases show models favor masculine cues.

Attention analysis reveals different responses to masculine and feminine cues.

Abstract

While gender bias in modern Neural Machine Translation (NMT) systems has received much attention, traditional evaluation metrics do not to fully capture the extent to which these systems integrate contextual gender cues. We propose a novel evaluation metric called Minimal Pair Accuracy (MPA), which measures the reliance of models on gender cues for gender disambiguation. MPA is designed to go beyond surface-level gender accuracy metrics by focusing on whether models adapt to gender cues in minimal pairs -- sentence pairs that differ solely in the gendered pronoun, namely the explicit indicator of the target's entity gender in the source language (EN). We evaluate a number of NMT models on the English-Italian (EN--IT) language pair using this metric, we show that they ignore available gender cues in most cases in favor of (statistical) stereotypical gender interpretation. We further show that in anti-stereotypical cases, these models tend to more consistently take masculine gender cues into account while ignoring the feminine cues. Furthermore, we analyze the attention head weights in the encoder component and show that while all models encode gender information to some extent, masculine cues elicit a more diffused response compared to the more concentrated and specialized responses to feminine gender cues.