Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation

TL;DR

This paper investigates gender bias in machine translation quality estimation metrics, revealing significant biases that favor masculine forms and impact downstream translation quality and data filtering.

Contribution

It introduces the first comprehensive analysis of gender bias in QE metrics and demonstrates its effects across multiple languages and domains.

Findings

Masculine translations score higher than feminine ones when gender is undisclosed.

Gender-neutral translations are penalized by biased QE metrics.

Context-aware QE metrics show more errors for feminine referents.

Abstract

Quality estimation (QE)-the automatic assessment of translation quality-has recently become crucial across several stages of the translation pipeline, from data curation to training and decoding. While QE metrics have been optimized to align with human judgments, whether they encode social biases has been largely overlooked. Biased QE risks favoring certain demographic groups over others, e.g., by exacerbating gaps in visibility and usability. This paper defines and investigates gender bias of QE metrics and discusses its downstream implications for machine translation (MT). Experiments with state-of-the-art QE metrics across multiple domains, datasets, and languages reveal significant bias. When a human entity's gender in the source is undisclosed, masculine-inflected translations score higher than feminine-inflected ones, and gender-neutral translations are penalized. Even when contextual cues disambiguate gender, using context-aware QE metrics leads to more errors in selecting the correct translation inflection for feminine referents than for masculine ones. Moreover, a biased QE metric affects data filtering and quality-aware decoding. Our findings underscore the need for a renewed focus on developing and evaluating QE metrics centered on gender.