It's not a Non-Issue: Negation as a Source of Error in Machine Translation

TL;DR

This paper investigates how negation affects the quality of machine translation, revealing that negation can cause significant errors and quality drops in multiple translation directions, highlighting an important challenge for MT systems.

Contribution

It provides a comprehensive analysis of negation errors in modern MT systems across 17 language pairs and offers linguistically motivated explanations for these errors.

Findings

Negation significantly impacts translation quality, with errors causing over 60% quality reduction.

Analysis reveals common patterns and causes of negation errors in MT.

Provides annotated data and code for further research on negation in MT.

Abstract

As machine translation (MT) systems progress at a rapid pace, questions of their adequacy linger. In this study we focus on negation, a universal, core property of human language that significantly affects the semantics of an utterance. We investigate whether translating negation is an issue for modern MT systems using 17 translation directions as test bed. Through thorough analysis, we find that indeed the presence of negation can significantly impact downstream quality, in some cases resulting in quality reductions of more than 60%. We also provide a linguistically motivated analysis that directly explains the majority of our findings. We release our annotations and code to replicate our analysis here: https://github.com/mosharafhossain/negation-mt.