Reinforcement Learning with Large Action Spaces for Neural Machine Translation

TL;DR

This paper investigates how large action spaces hinder reinforcement learning in neural machine translation and demonstrates that reducing the action space improves translation quality, with a novel initialization method yielding significant gains.

Contribution

The study shows that reducing the action space size enhances RL effectiveness in NMT and introduces a new initialization approach that generalizes over similar actions for better performance.

Findings

Reducing vocabulary size improves RL effectiveness.

Action space reduction via dimensionality techniques boosts BLEU scores.

Layer initialization with generalized actions yields 1.5 BLEU point improvement.

Abstract

Applying Reinforcement learning (RL) following maximum likelihood estimation (MLE) pre-training is a versatile method for enhancing neural machine translation (NMT) performance. However, recent work has argued that the gains produced by RL for NMT are mostly due to promoting tokens that have already received a fairly high probability in pre-training. We hypothesize that the large action space is a main obstacle to RL's effectiveness in MT, and conduct two sets of experiments that lend support to our hypothesis. First, we find that reducing the size of the vocabulary improves RL's effectiveness. Second, we find that effectively reducing the dimension of the action space without changing the vocabulary also yields notable improvement as evaluated by BLEU, semantic similarity, and human evaluation. Indeed, by initializing the network's final fully connected layer (that maps the network's internal dimension to the vocabulary dimension), with a layer that generalizes over similar actions, we obtain a substantial improvement in RL performance: 1.5 BLEU points on average.