Gender Bias in English-to-Greek Machine Translation

TL;DR

This paper examines gender bias in English-to-Greek machine translation systems, revealing persistent stereotypes and exploring GPT-4o's potential to mitigate bias by providing gender-inclusive alternatives.

Contribution

It introduces GendEL, a bilingual dataset for analyzing gender bias, and evaluates the bias in Google Translate, DeepL, and GPT-4o, highlighting their strengths and limitations.

Findings

Both MT systems show gender bias, especially in ambiguous cases.

DeepL outperforms Google Translate and GPT-4o in gender-unambiguous translations.

GPT-4o can generate gender-neutral and gendered alternatives, but residual bias persists.

Abstract

As the demand for inclusive language increases, concern has grown over the susceptibility of machine translation (MT) systems to reinforce gender stereotypes. This study investigates gender bias in two commercial MT systems, Google Translate and DeepL, focusing on the understudied English-to-Greek language pair. We address three aspects of gender bias: i) male bias, ii) occupational stereotyping, and iii) errors in anti-stereotypical translations. Additionally, we explore the potential of prompted GPT-4o as a bias mitigation tool that provides both gender-explicit and gender-neutral alternatives when necessary. To achieve this, we introduce GendEL, a manually crafted bilingual dataset of 240 gender-ambiguous and unambiguous sentences that feature stereotypical occupational nouns and adjectives. We find persistent gender bias in translations by both MT systems; while they perform well in cases where gender is explicitly defined, with DeepL outperforming both Google Translate and GPT-4o in feminine gender-unambiguous sentences, they are far from producing gender-inclusive or neutral translations when the gender is unspecified. GPT-4o shows promise, generating appropriate gendered and neutral alternatives for most ambiguous cases, though residual biases remain evident.