Gradient Descent in the Absence of Global Lipschitz Continuity of the Gradients

TL;DR

This paper provides a comprehensive global convergence analysis of gradient descent with diminishing step sizes for nonconvex functions with locally Lipschitz continuous gradients, addressing realistic data science scenarios.

Contribution

It introduces the first global convergence analysis of GD with diminishing step sizes under only local Lipschitz continuity, broadening understanding beyond classical assumptions.

Findings

Generalizes known results for GD with diminishing step sizes

Identifies behaviors in the divergence regime

Provides topological insights into convergence properties

Abstract

Gradient descent (GD) is a collection of continuous optimization methods that have achieved immeasurable success in practice. Owing to data science applications, GD with diminishing step sizes has become a prominent variant. While this variant of GD has been well-studied in the literature for objectives with globally Lipschitz continuous gradients or by requiring bounded iterates, objectives from data science problems do not satisfy such assumptions. Thus, in this work, we provide a novel global convergence analysis of GD with diminishing step sizes for differentiable nonconvex functions whose gradients are only locally Lipschitz continuous. Through our analysis, we generalize what is known about gradient descent with diminishing step sizes including interesting topological facts; and we elucidate the varied behaviors that can occur in the previously overlooked divergence regime. Thus, we provide the most general global convergence analysis of GD with diminishing step sizes under realistic conditions for data science problems.