Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regression

TL;DR

This paper demonstrates that using large stepsizes in gradient descent for regularized logistic regression accelerates convergence from exponential in the condition number to a square root rate, even with nonmonotonic objective evolution.

Contribution

It introduces the surprising result that large stepsizes can significantly speed up gradient descent for logistic regression, extending existing theory to strongly convex cases with finite minimizers.

Findings

Large stepsizes achieve $ ilde{O}( oot{ ext{condition number}})$ convergence.

Acceleration applies to minimizing population risk for separable data.

Characterizes the maximum stepsize for local and global convergence.

Abstract

We study gradient descent (GD) with a constant stepsize for $\ell_2$-regularized logistic regression with linearly separable data. Classical theory suggests small stepsizes to ensure monotonic reduction of the optimization objective, achieving exponential convergence in $\widetilde{\mathcal{O}}(\kappa)$ steps with $\kappa$ being the condition number. Surprisingly, we show that this can be accelerated to $\widetilde{\mathcal{O}}(\sqrt{\kappa})$ by simply using a large stepsize -- for which the objective evolves nonmonotonically. The acceleration brought by large stepsizes extends to minimizing the population risk for separable distributions, improving on the best-known upper bounds on the number of steps to reach a near-optimum. Finally, we characterize the largest stepsize for the local convergence of GD, which also determines the global convergence in special scenarios. Our results extend the analysis of Wu et al. (2024) from convex settings with minimizers at infinity to strongly convex cases with finite minimizers.