Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translation

TL;DR

This paper introduces auto-sizing, a method that dynamically adjusts Transformer architecture during training, leading to faster, more efficient, and better-performing low-resource machine translation models.

Contribution

It presents a novel auto-sizing approach that integrates architecture search into a single training run using regularization to prune neurons, improving efficiency and translation quality.

Findings

BLEU scores improved by up to 3.9 points on low-resource pairs

Model size reduced by one-third through neuron pruning

Auto-sizing enhances training efficiency and translation performance

Abstract

Neural sequence-to-sequence models, particularly the Transformer, are the state of the art in machine translation. Yet these neural networks are very sensitive to architecture and hyperparameter settings. Optimizing these settings by grid or random search is computationally expensive because it requires many training runs. In this paper, we incorporate architecture search into a single training run through auto-sizing, which uses regularization to delete neurons in a network over the course of training. On very low-resource language pairs, we show that auto-sizing can improve BLEU scores by up to 3.9 points while removing one-third of the parameters from the model.