Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpus

TL;DR

This paper introduces GeNTE, a new benchmark dataset for evaluating gender-neutral machine translation from English to Italian, and proposes a reference-free evaluation method to better assess inclusivity in translation.

Contribution

It presents the GeNTE benchmark dataset and explores automated evaluation methods specifically designed for gender-neutral translation, addressing limitations of existing approaches.

Findings

GeNTE is a natural, bilingual test set informed by user surveys.

Existing reference-based evaluation methods have limitations for gender-neutral translation.

A new reference-free evaluation approach is proposed for assessing gender-neutral translation quality.

Abstract

Gender inequality is embedded in our communication practices and perpetuated in translation technologies. This becomes particularly apparent when translating into grammatical gender languages, where machine translation (MT) often defaults to masculine and stereotypical representations by making undue binary gender assumptions. Our work addresses the rising demand for inclusive language by focusing head-on on gender-neutral translation from English to Italian. We start from the essentials: proposing a dedicated benchmark and exploring automated evaluation methods. First, we introduce GeNTE, a natural, bilingual test set for gender-neutral translation, whose creation was informed by a survey on the perception and use of neutral language. Based on GeNTE, we then overview existing reference-based evaluation approaches, highlight their limits, and propose a reference-free method more suitable to assess gender-neutral translation.