Reward Optimization for Neural Machine Translation with Learned Metrics

TL;DR

This paper explores optimizing neural machine translation models using learned, model-based metrics like BLEURT, demonstrating significant improvements in translation quality and human judgment compared to traditional BLEU-based training.

Contribution

It introduces a contrastive-margin loss for stable reward optimization with learned metrics and shows its effectiveness in improving translation quality over smoothed BLEU.

Findings

BLEURT-based training significantly increases metric scores.

Models trained with BLEURT show improved adequacy and coverage.

Human evaluations favor BLEURT-optimized models.

Abstract

Neural machine translation (NMT) models are conventionally trained with token-level negative log-likelihood (NLL), which does not guarantee that the generated translations will be optimized for a selected sequence-level evaluation metric. Multiple approaches are proposed to train NMT with BLEU as the reward, in order to directly improve the metric. However, it was reported that the gain in BLEU does not translate to real quality improvement, limiting the application in industry. Recently, it became clear to the community that BLEU has a low correlation with human judgment when dealing with state-of-the-art models. This leads to the emerging of model-based evaluation metrics. These new metrics are shown to have a much higher human correlation. In this paper, we investigate whether it is beneficial to optimize NMT models with the state-of-the-art model-based metric, BLEURT. We propose a contrastive-margin loss for fast and stable reward optimization suitable for large NMT models. In experiments, we perform automatic and human evaluations to compare models trained with smoothed BLEU and BLEURT to the baseline models. Results show that the reward optimization with BLEURT is able to increase the metric scores by a large margin, in contrast to limited gain when training with smoothed BLEU. The human evaluation shows that models trained with BLEURT improve adequacy and coverage of translations. Code is available via https://github.com/naver-ai/MetricMT.