Aligning Neural Machine Translation Models: Human Feedback in Training and Inference

TL;DR

This paper investigates how integrating human feedback-based reward models at various stages of neural machine translation can significantly improve translation quality, emphasizing data filtering, reinforcement learning, and reranking techniques.

Contribution

It provides a comprehensive comparison of methods for incorporating quality metrics into the MT pipeline, highlighting the importance of data filtering and combined approaches.

Findings

Effective data filtering based on quality estimates enhances RL in MT.

Combining RL training with reranking yields substantial quality improvements.

Unified approaches outperform individual techniques.

Abstract

Reinforcement learning from human feedback (RLHF) is a recent technique to improve the quality of the text generated by a language model, making it closer to what humans would generate. A core ingredient in RLHF's success in aligning and improving large language models (LLMs) is its reward model, trained using human feedback on model outputs. In machine translation (MT), where metrics trained from human annotations can readily be used as reward models, recent methods using minimum Bayes risk decoding and reranking have succeeded in improving the final quality of translation. In this study, we comprehensively explore and compare techniques for integrating quality metrics as reward models into the MT pipeline. This includes using the reward model for data filtering, during the training phase through RL, and at inference time by employing reranking techniques, and we assess the effects of combining these in a unified approach. Our experimental results, conducted across multiple translation tasks, underscore the crucial role of effective data filtering, based on estimated quality, in harnessing the full potential of RL in enhancing MT quality. Furthermore, our findings demonstrate the effectiveness of combining RL training with reranking techniques, showcasing substantial improvements in translation quality.