MQM-Chat: Multidimensional Quality Metrics for Chat Translation

TL;DR

This paper introduces MQM-Chat, a new evaluation metric designed to address the unique challenges of chat translation, highlighting its effectiveness in identifying errors and guiding improvements in stylized and dialogue-consistent translations.

Contribution

The study presents MQM-Chat, a novel multidimensional evaluation metric specifically tailored for chat translation, and demonstrates its utility across five different translation models.

Findings

All models exhibited fundamental errors in chat translation.

Different models showed specific shortcomings like omission and style loss.

MQM-Chat effectively evaluates chat translation quality, emphasizing stylized content and dialogue consistency.

Abstract

The complexities of chats pose significant challenges for machine translation models. Recognizing the need for a precise evaluation metric to address the issues of chat translation, this study introduces Multidimensional Quality Metrics for Chat Translation (MQM-Chat). Through the experiments of five models using MQM-Chat, we observed that all models generated certain fundamental errors, while each of them has different shortcomings, such as omission, overly correcting ambiguous source content, and buzzword issues, resulting in the loss of stylized information. Our findings underscore the effectiveness of MQM-Chat in evaluating chat translation, emphasizing the importance of stylized content and dialogue consistency for future studies.