Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation

TL;DR

This paper introduces MENT, a new dataset for evaluating non-literal translation quality, and proposes RATE, a novel framework that improves MT evaluation accuracy by addressing traditional metrics' limitations.

Contribution

The paper presents MENT, a curated dataset for non-literal translation evaluation, and introduces RATE, a reflective agentic framework that enhances evaluation reliability.

Findings

Traditional MT metrics are inaccurate for non-literal translations.

LLM-based evaluation methods face knowledge cutoff and inconsistency issues.

RATE improves correlation with human judgments by at least 3.2 points.

Abstract

Large Language Models (LLMs) have significantly advanced Machine Translation (MT), applying them to linguistically complex domains-such as Social Network Services, literature etc. In these scenarios, translations often require handling non-literal expressions, leading to the inaccuracy of MT metrics. To systematically investigate the reliability of MT metrics, we first curate a meta-evaluation dataset focused on non-literal translations, namely MENT. MENT encompasses four non-literal translation domains and features source sentences paired with translations from diverse MT systems, with 7,530 human-annotated scores on translation quality. Experimental results reveal the inaccuracies of traditional MT metrics and the limitations of LLM-as-a-Judge, particularly the knowledge cutoff and score inconsistency problem. To mitigate these limitations, we propose RATE, a novel agentic translation evaluation framework, centered by a reflective Core Agent that dynamically invokes specialized sub-agents. Experimental results indicate the efficacy of RATE, achieving an improvement of at least 3.2 points in combined system- and segment-level correlation with human judgments compared with current methods. Further experiments demonstrate the robustness of RATE to general-domain MT evaluation. Code and dataset are available at: https://github.com/BITHLP/RATE.