Thinking Outside the Ball: Optimal Learning with Gradient Descent for Generalized Linear Stochastic Convex Optimization

TL;DR

This paper demonstrates that early stopped gradient descent can achieve optimal sample complexity for generalized linear stochastic convex optimization without regularization, contrasting with traditional methods requiring more iterations.

Contribution

It introduces a novel analysis leveraging distribution-dependent uniform convergence to attain optimal learning rates with fewer iterations in generalized linear convex optimization.

Findings

Early stopping with gradient descent achieves optimal sample complexity.

Uniform convergence in distribution-dependent balls is key to improved efficiency.

Contrast with standard stochastic convex optimization requiring more iterations.

Abstract

We consider linear prediction with a convex Lipschitz loss, or more generally, stochastic convex optimization problems of generalized linear form, i.e.~where each instantaneous loss is a scalar convex function of a linear function. We show that in this setting, early stopped Gradient Descent (GD), without any explicit regularization or projection, ensures excess error at most $\epsilon$ (compared to the best possible with unit Euclidean norm) with an optimal, up to logarithmic factors, sample complexity of $\tilde{O}(1/\epsilon^2)$ and only $\tilde{O}(1/\epsilon^2)$ iterations. This contrasts with general stochastic convex optimization, where $\Omega(1/\epsilon^4)$ iterations are needed Amir et al. [2021b]. The lower iteration complexity is ensured by leveraging uniform convergence rather than stability. But instead of uniform convergence in a norm ball, which we show can guarantee suboptimal learning using $\Theta(1/\epsilon^4)$ samples, we rely on uniform convergence in a distribution-dependent ball.