Gender-specific Machine Translation with Large Language Models

TL;DR

This paper explores using large language models, specifically LLaMa, to generate gender-specific translations, demonstrating comparable accuracy to state-of-the-art systems and analyzing how coreference resolution influences gender bias and consistency.

Contribution

It introduces a method to produce gender-specific translations with LLaMa, highlighting the model's controllability and analyzing factors affecting gender bias and consistency.

Findings

LLaMa achieves translation accuracy comparable to NLLB.

Gender-specific translations depend on coreference resolution.

Higher gender variance in ambiguous datasets.

Abstract

While machine translation (MT) systems have seen significant improvements, it is still common for translations to reflect societal biases, such as gender bias. Decoder-only Large Language Models (LLMs) have demonstrated potential in MT, albeit with performance slightly lagging behind traditional encoder-decoder Neural Machine Translation (NMT) systems. However, LLMs offer a unique advantage: the ability to control the properties of the output through prompts. In this study, we leverage this flexibility to explore LLaMa's capability to produce gender-specific translations. Our results indicate that LLaMa can generate gender-specific translations with translation accuracy and gender bias comparable to NLLB, a state-of-the-art multilingual NMT system. Furthermore, our experiments reveal that LLaMa's gender-specific translations rely on coreference resolution to determine gender, showing higher gender variance in gender-ambiguous datasets but maintaining consistency in less ambiguous contexts. This research investigates the potential and challenges of using LLMs for gender-specific translations as an instance of the controllability of outputs offered by LLMs.