Normalized Direction-preserving Adam

TL;DR

This paper introduces ND-Adam, a variant of Adam that improves generalization in deep neural network training by controlling update directions and step sizes, bridging the gap between Adam and SGD.

Contribution

The paper proposes ND-Adam, a novel optimizer that enhances generalization by regulating update directions, and introduces regularization of softmax logits for classification tasks.

Findings

ND-Adam significantly improves generalization over standard Adam.

Regularizing softmax logits further enhances classification performance.

Bridging Adam and SGD offers insights into optimization and generalization.

Abstract

Adaptive optimization algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in some scenarios. However, recent studies show that they often lead to worse generalization performance than SGD, especially for training deep neural networks (DNNs). In this work, we identify the reasons that Adam generalizes worse than SGD, and develop a variant of Adam to eliminate the generalization gap. The proposed method, normalized direction-preserving Adam (ND-Adam), enables more precise control of the direction and step size for updating weight vectors, leading to significantly improved generalization performance. Following a similar rationale, we further improve the generalization performance in classification tasks by regularizing the softmax logits. By bridging the gap between SGD and Adam, we also hope to shed light on why certain optimization algorithms generalize better than others.