What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study

TL;DR

This study demonstrates that gender bias in machine translation causes tangible harms, such as increased effort and costs for women, highlighting the need for human-centered evaluations over automatic metrics.

Contribution

It provides the first extensive human-centered analysis linking gender bias in MT to real-world costs and shows current bias metrics are insufficient.

Findings

Feminine post-editing requires more effort and time.

Bias leads to higher financial costs for women.

Current automatic bias measures do not reflect actual disparities.

Abstract

Gender bias in machine translation (MT) is recognized as an issue that can harm people and society. And yet, advancements in the field rarely involve people, the final MT users, or inform how they might be impacted by biased technologies. Current evaluations are often restricted to automatic methods, which offer an opaque estimate of what the downstream impact of gender disparities might be. We conduct an extensive human-centered study to examine if and to what extent bias in MT brings harms with tangible costs, such as quality of service gaps across women and men. To this aim, we collect behavioral data from 90 participants, who post-edited MT outputs to ensure correct gender translation. Across multiple datasets, languages, and types of users, our study shows that feminine post-editing demands significantly more technical and temporal effort, also corresponding to higher financial costs. Existing bias measurements, however, fail to reflect the found disparities. Our findings advocate for human-centered approaches that can inform the societal impact of bias.