AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

TL;DR

AngularGrad is a novel optimizer for CNNs that leverages gradient angular information to adapt step sizes, resulting in smoother optimization and improved performance over existing methods.

Contribution

This paper introduces AngularGrad, the first optimizer to exploit gradient angular information for improved convergence in CNN training.

Findings

AngularGrad outperforms state-of-the-art optimizers on benchmark datasets.

Theoretical analysis shows AngularGrad has the same regret bound as Adam.

Two variants of AngularGrad using Tangent and Cosine functions are proposed.

Abstract

Convolutional neural networks (CNNs) are trained using stochastic gradient descent (SGD)-based optimizers. Recently, the adaptive moment estimation (Adam) optimizer has become very popular due to its adaptive momentum, which tackles the dying gradient problem of SGD. Nevertheless, existing optimizers are still unable to exploit the optimization curvature information efficiently. This paper proposes a new AngularGrad optimizer that considers the behavior of the direction/angle of consecutive gradients. This is the first attempt in the literature to exploit the gradient angular information apart from its magnitude. The proposed AngularGrad generates a score to control the step size based on the gradient angular information of previous iterations. Thus, the optimization steps become smoother as a more accurate step size of immediate past gradients is captured through the angular information. Two variants of AngularGrad are developed based on the use of Tangent or Cosine functions for computing the gradient angular information. Theoretically, AngularGrad exhibits the same regret bound as Adam for convergence purposes. Nevertheless, extensive experiments conducted on benchmark data sets against state-of-the-art methods reveal a superior performance of AngularGrad. The source code will be made publicly available at: https://github.com/mhaut/AngularGrad.