MMTE: Corpus and Metrics for Evaluating Machine Translation Quality of Metaphorical Language

TL;DR

This paper introduces a new corpus and evaluation metrics specifically designed to assess the quality of machine translation of figurative language, addressing a gap in existing evaluation methods.

Contribution

It presents a multilingual parallel metaphor corpus and human evaluation metrics focused on metaphor translation quality, covering aspects like equivalence, emotion, authenticity, and overall quality.

Findings

Figurative translations differ significantly from literal ones.

Existing MT evaluation methods overlook figurative language quality.

The proposed metrics effectively capture metaphor translation nuances.

Abstract

Machine Translation (MT) has developed rapidly since the release of Large Language Models and current MT evaluation is performed through comparison with reference human translations or by predicting quality scores from human-labeled data. However, these mainstream evaluation methods mainly focus on fluency and factual reliability, whilst paying little attention to figurative quality. In this paper, we investigate the figurative quality of MT and propose a set of human evaluation metrics focused on the translation of figurative language. We additionally present a multilingual parallel metaphor corpus generated by post-editing. Our evaluation protocol is designed to estimate four aspects of MT: Metaphorical Equivalence, Emotion, Authenticity, and Quality. In doing so, we observe that translations of figurative expressions display different traits from literal ones.