How to Measure Gender Bias in Machine Translation: Optimal Translators, Multiple Reference Points

TL;DR

This study systematically measures gender bias in Google Translate by comparing translations to an optimal non-biased translator, revealing prevalent bias against women and the influence of occupation-related words.

Contribution

It introduces a fair measure for gender bias in machine translation using multiple reference points and analyzes the impact of occupation and adjectives on bias.

Findings

Bias against women is more frequent in translations.

Translations align more with perception than occupational statistics.

Occupation words influence bias more than adjectives.

Abstract

In this paper, as a case study, we present a systematic study of gender bias in machine translation with Google Translate. We translated sentences containing names of occupations from Hungarian, a language with gender-neutral pronouns, into English. Our aim was to present a fair measure for bias by comparing the translations to an optimal non-biased translator. When assessing bias, we used the following reference points: (1) the distribution of men and women among occupations in both the source and the target language countries, as well as (2) the results of a Hungarian survey that examined if certain jobs are generally perceived as feminine or masculine. We also studied how expanding sentences with adjectives referring to occupations effect the gender of the translated pronouns. As a result, we found bias against both genders, but biased results against women are much more frequent. Translations are closer to our perception of occupations than to objective occupational statistics. Finally, occupations have a greater effect on translation than adjectives.