On Optimal Transformer Depth for Low-Resource Language Translation

TL;DR

This paper investigates the optimal depth of Transformer models for low-resource language translation, demonstrating that smaller models often outperform larger ones in this setting, thus reducing computational costs.

Contribution

The study reveals that low-resource NMT benefits from shallower Transformer models, challenging the trend of using very large models and promoting resource-efficient approaches.

Findings

Smaller Transformer models perform better in low-resource NMT.

Large models are more difficult to optimize and can hurt performance.

Resource-efficient models are effective for community-driven low-resource translation.

Abstract

Transformers have shown great promise as an approach to Neural Machine Translation (NMT) for low-resource languages. However, at the same time, transformer models remain difficult to optimize and require careful tuning of hyper-parameters to be useful in this setting. Many NMT toolkits come with a set of default hyper-parameters, which researchers and practitioners often adopt for the sake of convenience and avoiding tuning. These configurations, however, have been optimized for large-scale machine translation data sets with several millions of parallel sentences for European languages like English and French. In this work, we find that the current trend in the field to use very large models is detrimental for low-resource languages, since it makes training more difficult and hurts overall performance, confirming previous observations. We see our work as complementary to the Masakhane project ("Masakhane" means "We Build Together" in isiZulu.) In this spirit, low-resource NMT systems are now being built by the community who needs them the most. However, many in the community still have very limited access to the type of computational resources required for building extremely large models promoted by industrial research. Therefore, by showing that transformer models perform well (and often best) at low-to-moderate depth, we hope to convince fellow researchers to devote less computational resources, as well as time, to exploring overly large models during the development of these systems.