As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation

TL;DR

This paper introduces a comprehensive behavioral testing framework to evaluate neural machine translation systems' ability to accurately translate numerical data, revealing widespread issues and novel errors across various languages.

Contribution

It develops new test examples and assessment methods to expose numerical mistranslation in NMT systems, highlighting a general problem and proposing mitigation strategies.

Findings

Major commercial and research NMT systems fail on many numerical test cases.

Numerical mistranslation is prevalent across high- and low-resource languages.

The study uncovers previously unreported errors in NMT systems.

Abstract

Mistranslated numbers have the potential to cause serious effects, such as financial loss or medical misinformation. In this work we develop comprehensive assessments of the robustness of neural machine translation systems to numerical text via behavioural testing. We explore a variety of numerical translation capabilities a system is expected to exhibit and design effective test examples to expose system underperformance. We find that numerical mistranslation is a general issue: major commercial systems and state-of-the-art research models fail on many of our test examples, for high- and low-resource languages. Our tests reveal novel errors that have not previously been reported in NMT systems, to the best of our knowledge. Lastly, we discuss strategies to mitigate numerical mistranslation.