Translation Error Detection as Rationale Extraction

TL;DR

This paper demonstrates that feature attribution methods can effectively extract rationales from QE models to detect translation errors, introducing a semi-supervised word-level QE approach and a new interpretability benchmark.

Contribution

It introduces a novel semi-supervised method for word-level quality estimation and proposes using QE as a benchmark for evaluating explanation plausibility.

Findings

Rationales from QE models can identify translation errors.

Feature attribution explanations are interpretable to humans.

The proposed method improves word-level error detection.

Abstract

Recent Quality Estimation (QE) models based on multilingual pre-trained representations have achieved very competitive results when predicting the overall quality of translated sentences. Predicting translation errors, i.e. detecting specifically which words are incorrect, is a more challenging task, especially with limited amounts of training data. We hypothesize that, not unlike humans, successful QE models rely on translation errors to predict overall sentence quality. By exploring a set of feature attribution methods that assign relevance scores to the inputs to explain model predictions, we study the behaviour of state-of-the-art sentence-level QE models and show that explanations (i.e. rationales) extracted from these models can indeed be used to detect translation errors. We therefore (i) introduce a novel semi-supervised method for word-level QE and (ii) propose to use the QE task as a new benchmark for evaluating the plausibility of feature attribution, i.e. how interpretable model explanations are to humans.