What's the Difference Between Professional Human and Machine Translation? A Blind Multi-language Study on Domain-specific MT

TL;DR

This study compares professional human translation and machine translation in a blind multi-language evaluation, finding similar error types and post-editing efforts across most language pairs, highlighting the quality of MT.

Contribution

It provides a direct, blind comparison of HT and MT errors and editing effort across multiple languages, which was previously lacking.

Findings

Post-editing effort for MT is higher in only two out of three language pairs.

Number of segments with errors like wrong terminology, omissions, and typos is similar in HT and MT.

MT quality is comparable to professional human translation in many aspects.

Abstract

Machine translation (MT) has been shown to produce a number of errors that require human post-editing, but the extent to which professional human translation (HT) contains such errors has not yet been compared to MT. We compile pre-translated documents in which MT and HT are interleaved, and ask professional translators to flag errors and post-edit these documents in a blind evaluation. We find that the post-editing effort for MT segments is only higher in two out of three language pairs, and that the number of segments with wrong terminology, omissions, and typographical problems is similar in HT.