Adafactor: Adaptive Learning Rates with Sublinear Memory Cost

TL;DR

This paper introduces Adafactor, an optimizer that reduces memory usage by maintaining only row and column sums of second-moment estimates, while achieving comparable performance to Adam in training neural networks.

Contribution

The paper proposes a memory-efficient adaptive optimizer, Adafactor, with novel update schemes and scaling methods, enabling large-scale training with minimal auxiliary storage.

Findings

Adafactor matches Adam's performance in training Transformer models.

The method significantly reduces memory requirements for second-moment estimation.

Update clipping and decay schemes improve training stability.

Abstract

In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and per-column sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay rate of the second moment accumulator is too slow. We propose update clipping and a gradually increasing decay rate scheme as remedies. Combining these methods and dropping momentum, we achieve comparable results to the published Adam regime in training the Transformer model on the WMT 2014 English-German machine translation task, while using very little auxiliary storage in the optimizer. Finally, we propose scaling the parameter updates based on the scale of the parameters themselves.