Optimizing transformer-based machine translation model for single GPU training: a hyperparameter ablation study

TL;DR

This paper investigates hyperparameter effects on transformer-based machine translation models trained on a single GPU, revealing that larger models are not always better and identifying optimal configurations for cost-effective training.

Contribution

It provides a systematic ablation study showing how hyperparameter tuning can enable high-quality translation with smaller models on a single GPU.

Findings

Larger parameter counts do not guarantee better performance.

Optimal hyperparameter combinations can reduce model size without quality loss.

Training on a single GPU is feasible with careful hyperparameter selection.

Abstract

In machine translation tasks, the relationship between model complexity and performance is often presumed to be linear, driving an increase in the number of parameters and consequent demands for computational resources like multiple GPUs. To explore this assumption, this study systematically investigates the effects of hyperparameters through ablation on a sequence-to-sequence machine translation pipeline, utilizing a single NVIDIA A100 GPU. Contrary to expectations, our experiments reveal that combinations with the most parameters were not necessarily the most effective. This unexpected insight prompted a careful reduction in parameter sizes, uncovering "sweet spots" that enable training sophisticated models on a single GPU without compromising translation quality. The findings demonstrate an intricate relationship between hyperparameter selection, model size, and computational resource needs. The insights from this study contribute to the ongoing efforts to make machine translation more accessible and cost-effective, emphasizing the importance of precise hyperparameter tuning over mere scaling.