Optimal Asymptotic Rates for (Stochastic) Gradient Descent under the Local PL-Condition: A Geometric Approach

TL;DR

This paper demonstrates that stochastic gradient descent achieves optimal asymptotic convergence rates under the local PL-condition, even in non-convex settings inspired by neural networks, using a geometric analysis.

Contribution

It provides a geometric framework showing (S)GD's asymptotic rates match those of strongly convex quadratics under the local PL-condition.

Findings

Asymptotic convergence rate of (S)GD matches strongly convex quadratic rates.

Geometric interpretation of the PL-condition explains convergence behavior.

Results apply to non-convex functions satisfying the PL-inequality.

Abstract

Stochastic gradient descent (SGD) has been studied extensively over the past decades due to its simplicity and broad applicability in machine learning. In this work, we analyze the local behavior of gradient descent and stochastic gradient descent for minimizing $C^2$-functions that satisfy the Polyak-Lojasiewicz (PL) inequality and under a multiplicative gradient noise model motivated by overparameterized neural networks. Using a geometric interpretation of the PL-condition, we prove a simple yet surprising fact: in this possibly non-convex setting, the asymptotic convergence rate of (S)GD matches the rate obtained for strongly convex quadratics.