Gradient Descent-Type Methods: Background and Simple Unified Convergence Analysis

TL;DR

This chapter provides an elementary, unified convergence analysis of gradient descent and its variants, including stochastic and variance-reduced methods, emphasizing their mathematical structure and assumptions.

Contribution

It offers a simple, unified convergence framework for various gradient descent methods, including recent stochastic variants, based on elementary recursive analysis.

Findings

Unified convergence analysis for gradient descent variants

Illustration of analysis on common schemes

Clarification of assumptions and structures involved

Abstract

In this book chapter, we briefly describe the main components that constitute the gradient descent method and its accelerated and stochastic variants. We aim at explaining these components from a mathematical point of view, including theoretical and practical aspects, but at an elementary level. We will focus on basic variants of the gradient descent method and then extend our view to recent variants, especially variance-reduced stochastic gradient schemes (SGD). Our approach relies on revealing the structures presented inside the problem and the assumptions imposed on the objective function. Our convergence analysis unifies several known results and relies on a general, but elementary recursive expression. We have illustrated this analysis on several common schemes.