Early-Exit and Instant Confidence Translation Quality Estimation

TL;DR

This paper introduces cost-effective, uncertainty-aware quality estimation models for machine translation that enable early-exit computation and improve efficiency in evaluation and reranking tasks with minimal performance loss.

Contribution

The authors develop Instant Confidence COMET and Early-Exit COMET, novel models that reduce computational costs and provide uncertainty estimates for translation quality assessment.

Findings

Models reduce evaluation costs by 50% with minimal performance degradation.

Early-Exit COMET enables quality scoring at early model layers.

Instant Confidence COMET effectively estimates uncertainty at lower computational expense.

Abstract

Quality estimation is omnipresent in machine translation, for both evaluation and generation. Unfortunately, quality estimation models are often opaque and computationally expensive, making them impractical to be part of large-scale pipelines. In this work, we tackle two connected challenges: (1) reducing the cost of quality estimation at scale, and (2) developing an inexpensive uncertainty estimation method for quality estimation. To address the latter, we introduce Instant Confidence COMET, an uncertainty-aware quality estimation model that matches the performance of previous approaches at a fraction of their costs. We extend this to Early-Exit COMET, a quality estimation model that can compute quality scores and associated confidences already at early model layers, allowing us to early-exit computations and reduce evaluation costs. We also apply our model to machine translation reranking. We combine Early-Exit COMET with an upper confidence bound bandit algorithm to find the best candidate from a large pool without having to run the full evaluation model on all candidates. In both cases (evaluation and reranking) our methods reduce the required compute by 50% with very little degradation in performance. Finally, we show how Instant Confidence COMET can be used to decide which translations a human evaluator should score rather than relying on the COMET score.