It is Not as Good as You Think! Evaluating Simultaneous Machine Translation on Interpretation Data

TL;DR

This paper emphasizes the importance of training and evaluating simultaneous machine translation systems on real interpretation data, revealing significant performance gaps and proposing a data augmentation method to mitigate this issue.

Contribution

It introduces an interpretation test set, demonstrates the evaluation gap between translation and interpretation data, and proposes a T2I style transfer method to improve SiMT performance.

Findings

Evaluation gap of up to 13.83 BLEU score between translation and interpretation data.

T2I style transfer improves BLEU by up to 2.8 points.

Highlighting the need for large-scale interpretation corpora for better SiMT development.

Abstract

Most existing simultaneous machine translation (SiMT) systems are trained and evaluated on offline translation corpora. We argue that SiMT systems should be trained and tested on real interpretation data. To illustrate this argument, we propose an interpretation test set and conduct a realistic evaluation of SiMT trained on offline translations. Our results, on our test set along with 3 existing smaller scale language pairs, highlight the difference of up-to 13.83 BLEU score when SiMT models are evaluated on translation vs interpretation data. In the absence of interpretation training data, we propose a translation-to-interpretation (T2I) style transfer method which allows converting existing offline translations into interpretation-style data, leading to up-to 2.8 BLEU improvement. However, the evaluation gap remains notable, calling for constructing large-scale interpretation corpora better suited for evaluating and developing SiMT systems.