Findings of the WMT 2024 Shared Task on Chat Translation

TL;DR

This paper reports on the third edition of the Chat Translation Shared Task, analyzing system performance in translating bilingual customer support conversations across multiple language pairs with a focus on conversation context.

Contribution

It introduces new language pairs, provides comprehensive evaluation results, and highlights the importance of conversation context in translation quality assessment.

Findings

Systems excel at turn-level translation

Conversation-level translation quality needs improvement

Human evaluation is crucial for ranking systems

Abstract

This paper presents the findings from the third edition of the Chat Translation Shared Task. As with previous editions, the task involved translating bilingual customer support conversations, specifically focusing on the impact of conversation context in translation quality and evaluation. We also include two new language pairs: English-Korean and English-Dutch, in addition to the set of language pairs from previous editions: English-German, English-French, and English-Brazilian Portuguese. We received 22 primary submissions and 32 contrastive submissions from eight teams, with each language pair having participation from at least three teams. We evaluated the systems comprehensively using both automatic metrics and human judgments via a direct assessment framework. The official rankings for each language pair were determined based on human evaluation scores, considering performance in both translation directions--agent and customer. Our analysis shows that while the systems excelled at translating individual turns, there is room for improvement in overall conversation-level translation quality.