Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks

TL;DR

NovoGrad is a new adaptive optimization method for deep learning that normalizes gradients layer-wise, decouples weight decay, and offers robustness, efficiency, and competitive performance across various tasks.

Contribution

It introduces NovoGrad, an adaptive optimizer with layer-wise gradient normalization and decoupled weight decay, improving robustness and efficiency over existing methods.

Findings

Performs on par or better than SGD with momentum, Adam, and AdamW.

Robust to learning rate and initialization choices.

Effective in large batch training with reduced memory footprint.

Abstract

We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with momentum and Adam or AdamW. Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in a large batch setting, and (3) has two times smaller memory footprint than Adam.