Reinforcement Learning for Edit-Based Non-Autoregressive Neural Machine Translation

TL;DR

This paper applies reinforcement learning to edit-based non-autoregressive neural machine translation models, improving their performance by addressing decoding and training challenges, and investigates the effects of different RL strategies and temperature settings.

Contribution

It introduces RL methods to enhance edit-based NAR NMT models and analyzes the impact of training strategies and temperature parameters.

Findings

RL improves NAR model performance

Stepwise and episodic reward approaches have distinct advantages

Proper temperature setting is crucial for training effectiveness

Abstract

Non-autoregressive (NAR) language models are known for their low latency in neural machine translation (NMT). However, a performance gap exists between NAR and autoregressive models due to the large decoding space and difficulty in capturing dependency between target words accurately. Compounding this, preparing appropriate training data for NAR models is a non-trivial task, often exacerbating exposure bias. To address these challenges, we apply reinforcement learning (RL) to Levenshtein Transformer, a representative edit-based NAR model, demonstrating that RL with self-generated data can enhance the performance of edit-based NAR models. We explore two RL approaches: stepwise reward maximization and episodic reward maximization. We discuss the respective pros and cons of these two approaches and empirically verify them. Moreover, we experimentally investigate the impact of temperature setting on performance, confirming the importance of proper temperature setting for NAR models' training.