Exponential Convergence of (Stochastic) Gradient Descent for Separable Logistic Regression

TL;DR

This paper proves that increasing step sizes in gradient descent and stochastic gradient descent can achieve exponential convergence in separable logistic regression without instability, challenging the notion that acceleration requires edge-of-stability regimes.

Contribution

It introduces simple, non-adaptive step-size schedules that ensure exponential convergence within stable regimes, avoiding the need for complex adaptive methods or instability-based acceleration.

Findings

Gradient descent with increasing step sizes achieves exponential convergence.

Stochastic gradient descent also attains exponential rates with lightweight adaptive step sizes.

Stable, structured step-size growth suffices for acceleration in logistic regression.

Abstract

Gradient descent and stochastic gradient descent are central to modern machine learning, yet their behavior under large step sizes remains theoretically unclear. Recent work suggests that acceleration often arises near the edge of stability, where optimization trajectories become unstable and difficult to analyze. Existing results for separable logistic regression achieve faster convergence by explicitly leveraging such unstable regimes through constant or adaptive large step sizes. In this paper, we show that instability is not inherent to acceleration. We prove that gradient descent with a simple, non-adaptive increasing step-size schedule achieves exponential convergence for separable logistic regression under a margin condition, while remaining entirely within a stable optimization regime. The resulting method is anytime and does not require prior knowledge of the optimization horizon or target accuracy. We also establish exponential convergence of stochastic gradient descent using a lightweight adaptive step-size rule that avoids line search and specialized procedures, improving upon existing polynomial-rate guarantees. Together, our results demonstrate that carefully structured step-size growth alone suffices to obtain exponential acceleration for both gradient descent and stochastic gradient descent.