Evaluating Machine Translation Quality with Conformal Predictive Distributions

TL;DR

This paper introduces a conformal predictive distribution method to evaluate translation quality with reliable confidence scores, outperforming baselines across multiple language pairs.

Contribution

It presents a novel application of conformal predictive distributions for uncertainty estimation in machine translation quality assessment.

Findings

Outperforms baseline in coverage and sharpness on six language pairs

Guarantees prediction interval coverage at specified significance levels

Requires data exchangeability for optimal performance

Abstract

This paper presents a new approach for assessing uncertainty in machine translation by simultaneously evaluating translation quality and providing a reliable confidence score. Our approach utilizes conformal predictive distributions to produce prediction intervals with guaranteed coverage, meaning that for any given significance level $\epsilon$, we can expect the true quality score of a translation to fall out of the interval at a rate of $1-\epsilon$. In this paper, we demonstrate how our method outperforms a simple, but effective baseline on six different language pairs in terms of coverage and sharpness. Furthermore, we validate that our approach requires the data exchangeability assumption to hold for optimal performance.