Stopping Rules for Stochastic Gradient Descent via Anytime-Valid Confidence Sequences

TL;DR

This paper introduces a new framework of anytime-valid confidence sequences for stochastic gradient descent, enabling statistically valid, trajectory-dependent stopping rules that work in both convex and nonconvex optimization without prior horizon knowledge.

Contribution

It develops the first time-uniform, statistically valid stopping rules for SGD applicable to convex and nonconvex problems based only on observed trajectories.

Findings

Provides anytime-valid certificates for suboptimality in convex SGD.

Offers time-uniform stationarity certificates in nonconvex optimization.

Characterizes stopping-time complexity under standard stepsize schedules.

Abstract

The problem of stopping stochastic gradient descent (SGD) in an online manner, based solely on the observed trajectory, is a challenging theoretical problem with significant consequences for applications. While SGD is routinely monitored as it runs, the classical theory of SGD provides guarantees only at pre-specified iteration horizons and offers no valid way to decide, based on the observed trajectory, when further computation is justified. We address this longstanding gap by developing anytime-valid confidence sequences for stochastic gradient methods, which remain valid under continuous monitoring and directly induce statistically valid, trajectory-dependent stopping rules: stop as soon as the current upper confidence bound on an appropriate performance measure falls below a user-specified tolerance. The confidence sequences are constructed using nonnegative supermartingales, are time-uniform, and depend only on observable quantities along the SGD trajectory, without requiring prior knowledge of the optimization horizon. In convex optimization, this yields anytime-valid certificates for weighted suboptimality of projected SGD under general stepsize schedules, without assuming smoothness or strong convexity. In nonconvex optimization, it yields time-uniform certificates for weighted first-order stationarity under smoothness assumptions. We further characterize the stopping-time complexity of the resulting stopping rules under standard stepsize schedules. To the best of our knowledge, this is the first framework that provides statistically valid, time-uniform stopping rules for SGD across both convex and nonconvex settings based solely on its observed trajectory.