Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation

TL;DR

This paper explores the use of instruction-tuned large language models for document-level machine translation, demonstrating improved translation quality through direct prompting and proposing GPT-4-based evaluation methods that better capture document coherence and fluency.

Contribution

It introduces a simple prompting approach for document translation with LLMs and advocates for GPT-4-based evaluation over traditional BLEU scores.

Findings

Direct prompting improves document translation quality over sentence-by-sentence methods.

BLEU scores often do not reflect true document translation quality.

GPT-4-based evaluation provides a more nuanced assessment of translation coherence and fluency.

Abstract

Large language models (LLMs) have excelled in various NLP tasks, including machine translation (MT), yet most studies focus on sentence-level translation. This work investigates the inherent capability of instruction-tuned LLMs for document-level translation (docMT). Unlike prior approaches that require specialized techniques, we evaluate LLMs by directly prompting them to translate entire documents in a single pass. Our results show that this method improves translation quality compared to translating sentences separately, even without document-level fine-tuning. However, this advantage is not reflected in BLEU scores, which often favor sentence-based translations. We propose using the LLM-as-a-judge paradigm for evaluation, where GPT-4 is used to assess document coherence, accuracy, and fluency in a more nuanced way than n-gram-based metrics. Overall, our work demonstrates that instruction-tuned LLMs can effectively leverage document context for translation. However, we caution against using BLEU scores for evaluating docMT, as they often provide misleading outcomes, failing to capture the quality of document-level translation. Code and the outputs from GPT4-as-a-judge are available at https://github.com/EIT-NLP/BLEUless_DocMT