Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance

TL;DR

This paper investigates how translationese influences machine translation performance, revealing causal effects of training and testing data alignment and providing guidelines for better MT training and evaluation.

Contribution

It introduces CausalMT, a dataset with labeled translation directions, and analyzes causal impacts of data alignment factors on MT performance.

Findings

Train-test direction match significantly affects MT accuracy.

Data-model direction alignment has a large causal impact.

Test-model direction mismatch influences translation quality.

Abstract

Human-translated text displays distinct features from naturally written text in the same language. This phenomena, known as translationese, has been argued to confound the machine translation (MT) evaluation. Yet, we find that existing work on translationese neglects some important factors and the conclusions are mostly correlational but not causal. In this work, we collect CausalMT, a dataset where the MT training data are also labeled with the human translation directions. We inspect two critical factors, the train-test direction match (whether the human translation directions in the training and test sets are aligned), and data-model direction match (whether the model learns in the same direction as the human translation direction in the dataset). We show that these two factors have a large causal effect on the MT performance, in addition to the test-model direction mismatch highlighted by existing work on the impact of translationese. In light of our findings, we provide a set of suggestions for MT training and evaluation. Our code and data are at https://github.com/EdisonNi-hku/CausalMT