Learning Policies for Multilingual Training of Neural Machine Translation Systems

TL;DR

This paper introduces search-based and learned curricula for multilingual neural machine translation, enhancing low-resource translation performance by optimizing training data orderings and jointly learning curricula with the translation model.

Contribution

It proposes simple search-based curricula and a novel method to learn curricula from scratch using multi-arm bandits, improving low-resource MNMT performance.

Findings

Learned curricula provide better fine-tuning starting points.

Curricula improve overall translation accuracy on FLORES dataset.

Joint curriculum learning enhances low-resource translation quality.

Abstract

Low-resource Multilingual Neural Machine Translation (MNMT) is typically tasked with improving the translation performance on one or more language pairs with the aid of high-resource language pairs. In this paper, we propose two simple search based curricula -- orderings of the multilingual training data -- which help improve translation performance in conjunction with existing techniques such as fine-tuning. Additionally, we attempt to learn a curriculum for MNMT from scratch jointly with the training of the translation system with the aid of contextual multi-arm bandits. We show on the FLORES low-resource translation dataset that these learned curricula can provide better starting points for fine tuning and improve overall performance of the translation system.