Robust Training of Neural Networks Using Scale Invariant Architectures

TL;DR

This paper introduces a scale-invariant neural network architecture and training method that enables SGD to match the robustness and performance of adaptive optimizers like Adam, with added memory efficiency.

Contribution

The paper proposes a novel scale-invariant architecture and training procedure that allows SGD to achieve robustness comparable to adaptive methods, reducing memory usage.

Findings

Scale-invariant BERT (SIBERT) performs comparably to Adam-trained BERT.

The proposed method converges logarithmically with respect to initialization and loss scale.

Standard SGD may fail to converge under certain initializations.

Abstract

In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks, especially large language models. However, the use of adaptivity not only comes at the cost of extra memory but also raises the fundamental question: can non-adaptive methods like SGD enjoy similar benefits? In this paper, we provide an affirmative answer to this question by proposing to achieve both robust and memory-efficient training via the following general recipe: (1) modify the architecture and make it scale invariant, i.e. the scale of parameter doesn't affect the output of the network, (2) train with SGD and weight decay, and optionally (3) clip the global gradient norm proportional to weight norm multiplied by $\sqrt{\tfrac{2\lambda}{\eta}}$, where $\eta$ is learning rate and $\lambda$ is weight decay. We show that this general approach is robust to rescaling of parameter and loss by proving that its convergence only depends logarithmically on the scale of initialization and loss, whereas the standard SGD might not even converge for many initializations. Following our recipe, we design a scale invariant version of BERT, called SIBERT, which when trained simply by vanilla SGD achieves performance comparable to BERT trained by adaptive methods like Adam on downstream tasks.