Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations

TL;DR

This paper introduces the TWC dataset to evaluate gender bias and reasoning in machine translation, revealing widespread biases in current models and demonstrating that fine-tuning can significantly reduce these biases and improve translation quality.

Contribution

The paper presents the TWC dataset for assessing gender bias in translation and shows that fine-tuning models on this data reduces bias and enhances translation fairness and accuracy.

Findings

Models prefer masculine pronouns in gender-neutral contexts.

Fine-tuning on TWC reduces gender bias and stereotyping.

Open-source models can outperform proprietary systems after fine-tuning.

Abstract

Addressing gender bias and maintaining logical coherence in machine translation remains challenging, particularly when translating between natural gender languages, like English, and genderless languages, such as Persian, Indonesian, and Finnish. We introduce the Translate-with-Care (TWC) dataset, comprising 3,950 challenging scenarios across six low- to mid-resource languages, to assess translation systems' performance. Our analysis of diverse technologies, including GPT-4, mBART-50, NLLB-200, and Google Translate, reveals a universal struggle in translating genderless content, resulting in gender stereotyping and reasoning errors. All models preferred masculine pronouns when gender stereotypes could influence choices. Google Translate and GPT-4 showed particularly strong bias, favoring male pronouns 4-6 times more than feminine ones in leadership and professional success contexts. Fine-tuning mBART-50 on TWC substantially resolved these biases and errors, led to strong generalization, and surpassed proprietary LLMs while remaining open-source. This work emphasizes the need for targeted approaches to gender and semantic coherence in machine translation, particularly for genderless languages, contributing to more equitable and accurate translation systems.