Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency

TL;DR

This paper demonstrates that using a large, constant stepsize in gradient descent for logistic regression with linearly separable data leads to rapid initial oscillations followed by accelerated convergence, improving optimization efficiency without additional techniques.

Contribution

It shows that large stepsizes induce non-monotonic loss behavior that accelerates convergence, providing theoretical insights and versatile proof techniques applicable to various classification settings.

Findings

GD exits oscillatory phase in O(η) steps

Achieves O(1/(η t)) convergence after initial phase

Large stepsize T can lead to O(1/T^2) loss reduction

Abstract

We consider gradient descent (GD) with a constant stepsize applied to logistic regression with linearly separable data, where the constant stepsize $\eta$ is so large that the loss initially oscillates. We show that GD exits this initial oscillatory phase rapidly -- in $\mathcal{O}(\eta)$ steps -- and subsequently achieves an $\tilde{\mathcal{O}}(1 / (\eta t) )$ convergence rate after $t$ additional steps. Our results imply that, given a budget of $T$ steps, GD can achieve an accelerated loss of $\tilde{\mathcal{O}}(1/T^2)$ with an aggressive stepsize $\eta:= \Theta( T)$, without any use of momentum or variable stepsize schedulers. Our proof technique is versatile and also handles general classification loss functions (where exponential tails are needed for the $\tilde{\mathcal{O}}(1/T^2)$ acceleration), nonlinear predictors in the neural tangent kernel regime, and online stochastic gradient descent (SGD) with a large stepsize, under suitable separability conditions.