Optimizing Transformer for Low-Resource Neural Machine Translation

TL;DR

This paper investigates how to optimize Transformer models for low-resource neural machine translation, demonstrating that hyper-parameter tuning significantly enhances translation quality in data-scarce scenarios.

Contribution

It provides an analysis of Transformer performance under low-resource conditions and proposes optimized hyper-parameter settings to improve translation quality.

Findings

Optimized Transformer improves BLEU scores by up to 7.3 points.

Hyper-parameter tuning is crucial for low-resource NMT.

Performance varies significantly with different hyper-parameter configurations.

Abstract

Language pairs with limited amounts of parallel data, also known as low-resource languages, remain a challenge for neural machine translation. While the Transformer model has achieved significant improvements for many language pairs and has become the de facto mainstream architecture, its capability under low-resource conditions has not been fully investigated yet. Our experiments on different subsets of the IWSLT14 training data show that the effectiveness of Transformer under low-resource conditions is highly dependent on the hyper-parameter settings. Our experiments show that using an optimized Transformer for low-resource conditions improves the translation quality up to 7.3 BLEU points compared to using the Transformer default settings.