Context-Aware or Context-Insensitive? Assessing LLMs' Performance in Document-Level Translation

TL;DR

This paper evaluates how well large language models utilize document context in translation tasks, revealing limitations in pronoun translation and emphasizing the need for context-aware fine-tuning to enhance document-level translation quality.

Contribution

It provides a comprehensive analysis of LLMs' ability to leverage document context in translation, highlighting current shortcomings and proposing directions for improving context-awareness.

Findings

LLMs outperform encoder-decoder models in overall document translation

Pronoun translation remains challenging for LLMs despite overall improvements

Context-aware fine-tuning is necessary to improve LLMs' reliability in document translation

Abstract

Large language models (LLMs) are increasingly strong contenders in machine translation. In this work, we focus on document-level translation, where some words cannot be translated without context from outside the sentence. Specifically, we investigate the ability of prominent LLMs to utilize the document context during translation through a perturbation analysis (analyzing models' robustness to perturbed and randomized document context) and an attribution analysis (examining the contribution of relevant context to the translation). We conduct an extensive evaluation across nine LLMs from diverse model families and training paradigms, including translation-specialized LLMs, alongside two encoder-decoder transformer baselines. We find that LLMs' improved document-translation performance compared to encoder-decoder models is not reflected in pronoun translation performance. Our analysis highlight the need for context-aware finetuning of LLMs with a focus on relevant parts of the context to improve their reliability for document-level translation.