An Empirical Exploration of Curriculum Learning for Neural Machine Translation

TL;DR

This paper investigates how curriculum learning can improve the training efficiency of neural machine translation models by exploring different sample presentation strategies, showing potential for faster convergence without sacrificing translation quality.

Contribution

It provides an extensive empirical analysis of curriculum learning in neural machine translation, highlighting the importance of curriculum design choices.

Findings

Curriculum learning can speed up training convergence.

Sample difficulty criteria significantly affect results.

Proper hyperparameter tuning is crucial for success.

Abstract

Machine translation systems based on deep neural networks are expensive to train. Curriculum learning aims to address this issue by choosing the order in which samples are presented during training to help train better models faster. We adopt a probabilistic view of curriculum learning, which lets us flexibly evaluate the impact of curricula design, and perform an extensive exploration on a German-English translation task. Results show that it is possible to improve convergence time at no loss in translation quality. However, results are highly sensitive to the choice of sample difficulty criteria, curriculum schedule and other hyperparameters.