Disentangling Uncertainty in Machine Translation Evaluation

TL;DR

This paper introduces advanced uncertainty prediction methods for machine translation evaluation metrics, improving interpretability and reliability, especially under noisy or out-of-domain conditions, while reducing computational costs.

Contribution

It proposes new training objectives for the COMET metric to better predict different types of uncertainty, surpassing existing techniques like Monte Carlo dropout and deep ensembles.

Findings

Enhanced uncertainty prediction accuracy on WMT datasets

Significant reduction in computational costs

Ability to distinguish between different uncertainty sources

Abstract

Trainable evaluation metrics for machine translation (MT) exhibit strong correlation with human judgements, but they are often hard to interpret and might produce unreliable scores under noisy or out-of-domain data. Recent work has attempted to mitigate this with simple uncertainty quantification techniques (Monte Carlo dropout and deep ensembles), however these techniques (as we show) are limited in several ways -- for example, they are unable to distinguish between different kinds of uncertainty, and they are time and memory consuming. In this paper, we propose more powerful and efficient uncertainty predictors for MT evaluation, and we assess their ability to target different sources of aleatoric and epistemic uncertainty. To this end, we develop and compare training objectives for the COMET metric to enhance it with an uncertainty prediction output, including heteroscedastic regression, divergence minimization, and direct uncertainty prediction. Our experiments show improved results on uncertainty prediction for the WMT metrics task datasets, with a substantial reduction in computational costs. Moreover, they demonstrate the ability of these predictors to address specific uncertainty causes in MT evaluation, such as low quality references and out-of-domain data.