Tricks for Training Sparse Translation Models

TL;DR

This paper explores techniques to improve training of sparse multilingual translation models, addressing data imbalance issues with simple methods like temperature heating and dense pre-training, leading to faster convergence and better performance.

Contribution

It introduces two straightforward techniques—temperature heating and dense pre-training—to enhance sparse multilingual translation models, demonstrating significant performance gains and faster convergence.

Findings

Improved translation performance on multilingual benchmarks.

More than 2x faster model convergence with combined methods.

Sparse architectures can underperform without specific mitigation techniques.

Abstract

Multi-task learning with an unbalanced data distribution skews model learning towards high resource tasks, especially when model capacity is fixed and fully shared across all tasks. Sparse scaling architectures, such as BASELayers, provide flexible mechanisms for different tasks to have a variable number of parameters, which can be useful to counterbalance skewed data distributions. We find that that sparse architectures for multilingual machine translation can perform poorly out of the box, and propose two straightforward techniques to mitigate this - a temperature heating mechanism and dense pre-training. Overall, these methods improve performance on two multilingual translation benchmarks compared to standard BASELayers and Dense scaling baselines, and in combination, more than 2x model convergence speed.