Local Quadratic Convergence of Stochastic Gradient Descent with Adaptive Step Size

TL;DR

This paper proves that stochastic gradient descent with adaptive step size can achieve local quadratic convergence for certain problems, enhancing understanding of its efficiency in practical machine learning tasks.

Contribution

It is the first to establish local quadratic convergence of adaptive stochastic gradient descent methods for specific problems like matrix inversion.

Findings

SGD with adaptive step size achieves local quadratic convergence.

Theoretical results apply to problems such as matrix inversion.

Enhances understanding of convergence behavior in adaptive stochastic optimization.

Abstract

Establishing a fast rate of convergence for optimization methods is crucial to their applicability in practice. With the increasing popularity of deep learning over the past decade, stochastic gradient descent and its adaptive variants (e.g. Adagrad, Adam, etc.) have become prominent methods of choice for machine learning practitioners. While a large number of works have demonstrated that these first order optimization methods can achieve sub-linear or linear convergence, we establish local quadratic convergence for stochastic gradient descent with adaptive step size for problems such as matrix inversion.