Optimizing the Training Schedule of Multilingual NMT using Reinforcement Learning

TL;DR

This paper introduces reinforcement learning algorithms to optimize the training schedule of multilingual neural machine translation, significantly improving translation quality for low-resource languages by intelligently ordering language presentations.

Contribution

It proposes two reinforcement learning methods, Teacher-Student Curriculum Learning and Deep Q Network, to optimize multilingual NMT training schedules, a novel approach in this context.

Findings

Deep Q Network improves BLEU and COMET scores

Optimized schedules outperform random and shuffled baselines

Effective adjustment of language presentation frequency

Abstract

Multilingual NMT is a viable solution for translating low-resource languages (LRLs) when data from high-resource languages (HRLs) from the same language family is available. However, the training schedule, i.e. the order of presentation of languages, has an impact on the quality of such systems. Here, in a many-to-one translation setting, we propose to apply two algorithms that use reinforcement learning to optimize the training schedule of NMT: (1) Teacher-Student Curriculum Learning and (2) Deep Q Network. The former uses an exponentially smoothed estimate of the returns of each action based on the loss on monolingual or multilingual development subsets, while the latter estimates rewards using an additional neural network trained from the history of actions selected in different states of the system, together with the rewards received. On a 8-to-1 translation dataset with LRLs and HRLs, our second method improves BLEU and COMET scores with respect to both random selection of monolingual batches and shuffled multilingual batches, by adjusting the number of presentations of LRL vs. HRL batches.