High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics

TL;DR

This paper challenges the assumption that the highest probability translation is the best, proposing Minimum Bayes Risk decoding with neural metrics to improve translation quality beyond traditional beam search results.

Contribution

It introduces MBR decoding using neural metrics like BLEURT to select higher quality translations that are not necessarily the most probable, improving human evaluation scores.

Findings

MBR decoding with neural metrics outperforms beam search in human evaluations.

Lower-probability translations can have higher quality according to neural metrics.

Using neural metrics for decoding leads to more diverse and higher-quality translations.

Abstract

In Neural Machine Translation, it is typically assumed that the sentence with the highest estimated probability should also be the translation with the highest quality as measured by humans. In this work, we question this assumption and show that model estimates and translation quality only vaguely correlate. We apply Minimum Bayes Risk (MBR) decoding on unbiased samples to optimize diverse automated metrics of translation quality as an alternative inference strategy to beam search. Instead of targeting the hypotheses with the highest model probability, MBR decoding extracts the hypotheses with the highest estimated quality. Our experiments show that the combination of a neural translation model with a neural reference-based metric, BLEURT, results in significant improvement in human evaluations. This improvement is obtained with translations different from classical beam-search output: these translations have much lower model likelihood and are less favored by surface metrics like BLEU.