Small Batch Sizes Improve Training of Low-Resource Neural MT

TL;DR

This paper demonstrates that in low-resource neural machine translation, smaller batch sizes lead to better performance and faster training, challenging the common practice of using large batch sizes for efficiency.

Contribution

The study provides theoretical and experimental evidence that smaller batch sizes improve training outcomes in low-resource NMT, highlighting their regularization benefits.

Findings

Smaller batch sizes yield higher translation scores.

Training with smaller batches is faster in low-resource settings.

Large batch sizes do not necessarily improve performance.

Abstract

We study the role of an essential hyper-parameter that governs the training of Transformers for neural machine translation in a low-resource setting: the batch size. Using theoretical insights and experimental evidence, we argue against the widespread belief that batch size should be set as large as allowed by the memory of the GPUs. We show that in a low-resource setting, a smaller batch size leads to higher scores in a shorter training time, and argue that this is due to better regularization of the gradients during training.