A Multi-task Learning Framework for Evaluating Machine Translation of Emotion-loaded User-generated Content

TL;DR

This paper introduces a multi-task learning framework that evaluates machine translation quality of emotion-rich user-generated content by jointly assessing translation errors and emotion classification, achieving state-of-the-art results.

Contribution

The paper presents a novel multi-task architecture with a combined loss function for simultaneous translation quality assessment and emotion classification in UGC.

Findings

Achieves state-of-the-art performance on UGC translation evaluation

Demonstrates improved generalization over multiple datasets

Provides comprehensive analysis of MT evaluation for emotion-loaded content

Abstract

Machine translation (MT) of user-generated content (UGC) poses unique challenges, including handling slang, emotion, and literary devices like irony and sarcasm. Evaluating the quality of these translations is challenging as current metrics do not focus on these ubiquitous features of UGC. To address this issue, we utilize an existing emotion-related dataset that includes emotion labels and human-annotated translation errors based on Multi-dimensional Quality Metrics. We extend it with sentence-level evaluation scores and word-level labels, leading to a dataset suitable for sentence- and word-level translation evaluation and emotion classification, in a multi-task setting. We propose a new architecture to perform these tasks concurrently, with a novel combined loss function, which integrates different loss heuristics, like the Nash and Aligned losses. Our evaluation compares existing fine-tuning and multi-task learning approaches, assessing generalization with ablative experiments over multiple datasets. Our approach achieves state-of-the-art performance and we present a comprehensive analysis for MT evaluation of UGC.