Document-Level Machine Translation with Large Language Models

TL;DR

This paper evaluates large language models like GPT-3.5 and GPT-4 for document-level machine translation, showing they outperform commercial systems and possess strong discourse modeling abilities, highlighting their potential as a new translation paradigm.

Contribution

The study provides an in-depth evaluation of LLMs for document-level MT, comparing their performance with existing systems and analyzing their discourse modeling capabilities.

Findings

GPT-3.5 and GPT-4 outperform commercial MT systems in human evaluations.

GPT-4 has stronger linguistic knowledge probing abilities than GPT-3.5.

LLMs demonstrate significant potential for discourse-aware document translation.

Abstract

Large language models (LLMs) such as ChatGPT can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. Taking document-level machine translation (MT) as a testbed, this paper provides an in-depth evaluation of LLMs' ability on discourse modeling. The study focuses on three aspects: 1) Effects of Context-Aware Prompts, where we investigate the impact of different prompts on document-level translation quality and discourse phenomena; 2) Comparison of Translation Models, where we compare the translation performance of ChatGPT with commercial MT systems and advanced document-level MT methods; 3) Analysis of Discourse Modelling Abilities, where we further probe discourse knowledge encoded in LLMs and shed light on impacts of training techniques on discourse modeling. By evaluating on a number of benchmarks, we surprisingly find that LLMs have demonstrated superior performance and show potential to become a new paradigm for document-level translation: 1) leveraging their powerful long-text modeling capabilities, GPT-3.5 and GPT-4 outperform commercial MT systems in terms of human evaluation; 2) GPT-4 demonstrates a stronger ability for probing linguistic knowledge than GPT-3.5. This work highlights the challenges and opportunities of LLMs for MT, which we hope can inspire the future design and evaluation of LLMs.We release our data and annotations at https://github.com/longyuewangdcu/Document-MT-LLM.