As Little as Possible, as Much as Necessary: Detecting Over- and Undertranslations with Contrastive Conditioning

TL;DR

This paper introduces a contrastive conditioning method to detect over- and undertranslations in neural machine translation, effectively identifying content omissions and additions without needing reference translations.

Contribution

It presents a novel, reference-free approach using off-the-shelf models to identify translation errors through likelihood comparisons.

Findings

Comparable accuracy to supervised quality estimation methods

Effective detection of superfluous words and untranslated content

Works without reference translations

Abstract

Omission and addition of content is a typical issue in neural machine translation. We propose a method for detecting such phenomena with off-the-shelf translation models. Using contrastive conditioning, we compare the likelihood of a full sequence under a translation model to the likelihood of its parts, given the corresponding source or target sequence. This allows to pinpoint superfluous words in the translation and untranslated words in the source even in the absence of a reference translation. The accuracy of our method is comparable to a supervised method that requires a custom quality estimation model.