A Generalized Version of Chung's Lemma and its Applications

TL;DR

This paper introduces a generalized Chung's Lemma that provides a unified, non-asymptotic framework for analyzing convergence rates of stochastic optimization methods under various step size strategies and conditions.

Contribution

It extends Chung's Lemma to a broader class of step sizes, enabling tight convergence analysis for stochastic methods under general conditions, including the -Polyak-Lojasiewicz condition.

Findings

Exponential step sizes adaptively achieve optimal convergence rates.

New non-asymptotic complexity results for SGD and RR under general conditions.

The generalized lemma simplifies convergence proofs across diverse step size rules.

Abstract

Chung's Lemma is a classical tool for establishing asymptotic convergence rates of (stochastic) optimization methods under strong convexity-type assumptions and appropriate polynomial diminishing step sizes. In this work, we develop a generalized version of Chung's Lemma, which provides a simple non-asymptotic convergence framework for a more general family of step size rules. We demonstrate broad applicability of the proposed generalized lemma by deriving tight non-asymptotic convergence rates for a large variety of stochastic methods. In particular, we obtain partially new non-asymptotic complexity results for stochastic optimization methods, such as Stochastic Gradient Descent (SGD) and Random Reshuffling (RR), under a general $(\theta,\mu)$-Polyak-Lojasiewicz (PL) condition and for various step sizes strategies, including polynomial, constant, exponential, and cosine step sizes rules. Notably, as a by-product of our analysis, we observe that exponential step sizes exhibit superior adaptivity to both landscape geometry and gradient noise; specifically, they achieve optimal convergence rates without requiring exact knowledge of the underlying landscape or separate parameter selection strategies for noisy and noise-free regimes. Our results demonstrate that the developed variant of Chung's Lemma offers a versatile, systematic, and streamlined approach to establish non-asymptotic convergence rates under general step size rules.