A Robust Adaptive Stochastic Gradient Method for Deep Learning

TL;DR

This paper introduces a new adaptive stochastic gradient method that uses curvature information and variance reduction to improve convergence and performance in deep learning models.

Contribution

It proposes an innovative adaptive learning rate algorithm leveraging stochastic curvature estimates and a novel variance reduction technique for enhanced deep learning optimization.

Findings

Achieved better performance than popular SGD variants in neural network training.

Demonstrated faster convergence with the proposed adaptive method.

Improved robustness to noise in stochastic gradient estimates.

Abstract

Stochastic gradient algorithms are the main focus of large-scale optimization problems and led to important successes in the recent advancement of the deep learning algorithms. The convergence of SGD depends on the careful choice of learning rate and the amount of the noise in stochastic estimates of the gradients. In this paper, we propose an adaptive learning rate algorithm, which utilizes stochastic curvature information of the loss function for automatically tuning the learning rates. The information about the element-wise curvature of the loss function is estimated from the local statistics of the stochastic first order gradients. We further propose a new variance reduction technique to speed up the convergence. In our experiments with deep neural networks, we obtained better performance compared to the popular stochastic gradient algorithms.