Understanding the Impact of UGC Specificities on Translation Quality

TL;DR

This paper critically examines the challenges of evaluating user-generated content (UGC) translation quality, introduces a new annotated dataset, and analyzes how UGC specificities affect translation performance more precisely.

Contribution

It introduces a novel annotated dataset for UGC translation evaluation and provides detailed analysis of how UGC specificities impact translation quality.

Findings

Standard metrics are insufficient for UGC translation evaluation.

UGC specificities significantly affect translation quality.

The new dataset enables more precise impact measurement.

Abstract

This work takes a critical look at the evaluation of user-generated content automatic translation, the well-known specificities of which raise many challenges for MT. Our analyses show that measuring the average-case performance using a standard metric on a UGC test set falls far short of giving a reliable image of the UGC translation quality. That is why we introduce a new data set for the evaluation of UGC translation in which UGC specificities have been manually annotated using a fine-grained typology. Using this data set, we conduct several experiments to measure the impact of different kinds of UGC specificities on translation quality, more precisely than previously possible.