Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models

TL;DR

This paper investigates how input length affects large language model-based machine translation evaluation and proposes strategies to mitigate length bias, improving reliability for long document assessments.

Contribution

It identifies the impact of input length on LLM evaluation accuracy and introduces prompting and fine-tuning methods to reduce this bias.

Findings

Longer texts lead to fewer error spans detected.

Granularity-aligned prompting improves evaluation consistency.

Fine-tuning further mitigates length bias.

Abstract

Accurately evaluating machine-translated text remains a long-standing challenge, particularly for long documents. Recent work has shown that large language models (LLMs) can serve as reliable and interpretable sentence-level translation evaluators via MQM error span annotations. With modern LLMs supporting larger context windows, a natural question arises: can we feed entire document translations into an LLM for quality assessment? Ideally, evaluation should be invariant to text length, producing consistent error spans regardless of input granularity. However, our analysis shows that text length significantly impacts evaluation: longer texts lead to fewer error spans and reduced system ranking accuracy. To address this limitation, we evaluate several strategies, including granularity-aligned prompting, Focus Sentence Prompting (FSP), and a fine-tuning approach to better align LLMs with the evaluation task. The latter two methods largely mitigate this length bias, making LLMs more reliable for long-form translation evaluation.