SALTED: A Framework for SAlient Long-Tail Translation Error Detection

TL;DR

SALTED is a framework that detects rare but critical long-tail errors in machine translation, improving transparency and enabling targeted model improvements.

Contribution

It introduces a specifications-based testing framework with high-precision error detectors for identifying long-tail translation errors.

Findings

Detectors effectively identify salient long-tail errors.

Framework enables filtering, fixing, and testing of MT models.

Improves visibility into rare translation failures.

Abstract

Traditional machine translation (MT) metrics provide an average measure of translation quality that is insensitive to the long tail of behavioral problems in MT. Examples include translation of numbers, physical units, dropped content and hallucinations. These errors, which occur rarely and unpredictably in Neural Machine Translation (NMT), greatly undermine the reliability of state-of-the-art MT systems. Consequently, it is important to have visibility into these problems during model development. Towards this direction, we introduce SALTED, a specifications-based framework for behavioral testing of MT models that provides fine-grained views of salient long-tail errors, permitting trustworthy visibility into previously invisible problems. At the core of our approach is the development of high-precision detectors that flag errors (or alternatively, verify output correctness) between a source sentence and a system output. We demonstrate that such detectors could be used not just to identify salient long-tail errors in MT systems, but also for higher-recall filtering of the training data, fixing targeted errors with model fine-tuning in NMT and generating novel data for metamorphic testing to elicit further bugs in models.