Quality-Aware Decoding for Neural Machine Translation

TL;DR

This paper introduces a quality-aware decoding method for neural machine translation that integrates recent evaluation metrics, leading to improved translation quality over traditional MAP decoding, as demonstrated across multiple datasets and models.

Contribution

It proposes a novel decoding approach that incorporates evaluation metrics into the decoding process, bridging the gap between quality estimation and translation generation.

Findings

Quality-aware decoding outperforms MAP decoding according to automatic metrics.

It achieves better human assessment scores.

The method is effective across various datasets and model types.

Abstract

Despite the progress in machine translation quality estimation and evaluation in the last years, decoding in neural machine translation (NMT) is mostly oblivious to this and centers around finding the most probable translation according to the model (MAP decoding), approximated with beam search. In this paper, we bring together these two lines of research and propose quality-aware decoding for NMT, by leveraging recent breakthroughs in reference-free and reference-based MT evaluation through various inference methods like $N$-best reranking and minimum Bayes risk decoding. We perform an extensive comparison of various possible candidate generation and ranking methods across four datasets and two model classes and find that quality-aware decoding consistently outperforms MAP-based decoding according both to state-of-the-art automatic metrics (COMET and BLEURT) and to human assessments. Our code is available at https://github.com/deep-spin/qaware-decode.