DEAM: Adaptive Momentum with Discriminative Weight for Stochastic Optimization

TL;DR

DEAM introduces an adaptive momentum optimization algorithm that automatically adjusts momentum weights based on discriminative angles, reducing hyperparameters and improving convergence speed in deep learning training.

Contribution

The paper proposes DEAM, a novel optimizer with automatic momentum weight computation and a backtrack term, enhancing convergence over existing methods like ADAM.

Findings

DEAM converges faster than ADAM and SGD in experiments.

DEAM performs well on both convex and non-convex deep learning tasks.

DEAM reduces the need for hyperparameter tuning.

Abstract

Optimization algorithms with momentum, e.g., (ADAM), have been widely used for building deep learning models due to the faster convergence rates compared with stochastic gradient descent (SGD). Momentum helps accelerate SGD in the relevant directions in parameter updating, which can minify the oscillations of parameters update route. However, there exist errors in some update steps in optimization algorithms with momentum like ADAM. The fixed momentum weight (e.g., \beta_1 in ADAM) will propagate errors in momentum computing. In this paper, we introduce a novel optimization algorithm, namely Discriminative wEight on Adaptive Momentum (DEAM). Instead of assigning the momentum term weight with a fixed hyperparameter, DEAM proposes to compute the momentum weight automatically based on the discriminative angle. In this way, DEAM involves fewer hyperparameters. DEAM also contains a novel backtrack term, which restricts redundant updates when the correction of the last step is needed. Extensive experiments demonstrate that DEAM can achieve a faster convergence rate than the existing optimization algorithms in training the deep learning models of both convex and non-convex situations.