AdaL: Adaptive Gradient Transformation Contributes to Convergences and Generalizations

TL;DR

AdaL is an adaptive gradient transformation method that improves convergence speed and enhances generalization in deep learning by modifying the gradient dynamics throughout training.

Contribution

The paper introduces AdaL, a novel adaptive gradient method that accelerates convergence and improves generalization by transforming gradients to control noise and oscillations.

Findings

AdaL accelerates convergence in deep learning models.

AdaL achieves better generalization compared to traditional adaptive methods.

Theoretical proof of convergence for AdaL.

Abstract

Adaptive optimization methods have been widely used in deep learning. They scale the learning rates adaptively according to the past gradient, which has been shown to be effective to accelerate the convergence. However, they suffer from poor generalization performance compared with SGD. Recent studies point that smoothing exponential gradient noise leads to generalization degeneration phenomenon. Inspired by this, we propose AdaL, with a transformation on the original gradient. AdaL accelerates the convergence by amplifying the gradient in the early stage, as well as dampens the oscillation and stabilizes the optimization by shrinking the gradient later. Such modification alleviates the smoothness of gradient noise, which produces better generalization performance. We have theoretically proved the convergence of AdaL and demonstrated its effectiveness on several benchmarks.