Towards Explainable Evaluation Metrics for Machine Translation

TL;DR

This paper reviews the importance of explainability in machine translation evaluation metrics, analyzing recent techniques and proposing future directions for transparent, high-quality assessment methods.

Contribution

It provides a comprehensive synthesis of explainable evaluation metrics, discusses state-of-the-art approaches, and outlines a vision for next-generation explainable metrics including natural language explanations.

Findings

Classical metrics remain dominant due to transparency.

Recent models like ChatGPT and GPT-4 are being integrated into explainable metrics.

The paper proposes future directions for natural language explanations in evaluation.

Abstract

Unlike classical lexical overlap metrics such as BLEU, most current evaluation metrics for machine translation (for example, COMET or BERTScore) are based on black-box large language models. They often achieve strong correlations with human judgments, but recent research indicates that the lower-quality classical metrics remain dominant, one of the potential reasons being that their decision processes are more transparent. To foster more widespread acceptance of novel high-quality metrics, explainability thus becomes crucial. In this concept paper, we identify key properties as well as key goals of explainable machine translation metrics and provide a comprehensive synthesis of recent techniques, relating them to our established goals and properties. In this context, we also discuss the latest state-of-the-art approaches to explainable metrics based on generative models such as ChatGPT and GPT4. Finally, we contribute a vision of next-generation approaches, including natural language explanations. We hope that our work can help catalyze and guide future research on explainable evaluation metrics and, mediately, also contribute to better and more transparent machine translation systems.