Analysis of Schedule-Free Nonconvex Optimization

TL;DR

This paper develops a Lyapunov-based analysis for schedule-free first-order methods in nonconvex optimization, providing horizon-independent convergence guarantees and validating them through numerical experiments.

Contribution

It introduces a robust Lyapunov framework that extends schedule-free methods' convergence guarantees to nonconvex settings with minimal assumptions.

Findings

Horizon-agnostic convergence bounds for nonconvex SF methods.

Validation of theoretical rates through PEP experiments.

Potential for tighter bounds on the original SF algorithm.

Abstract

First-order methods underpin most large-scale learning algorithms, yet their classical convergence guarantees hinge on carefully scheduled step-sizes that depend on the total horizon $T$, which is rarely known in advance. The Schedule-Free (SF) method promises optimal performance with hyperparameters that are independent of $T$ by interpolating between Polyak--Ruppert averaging and momentum, but nonconvex analysis of SF has been limited or reliant on strong global assumptions. We introduce a robust Lyapunov framework that, under only $L$-smoothness and lower-boundedness, reduces SF analysis to a single-step descent inequality. This yields horizon-agnostic bounds in the nonconvex setting: $O(1/\log T)$ for constant step + PR averaging, $O(\log T/T)$ for a linearly growing step-size, and a continuum of $O(T^{-(1-\alpha)})$ rates for polynomial averaging. We complement these proofs with Performance Estimation Problem (PEP) experiments that numerically validate our rates and suggest that our $O(1/\log T)$ bound on the original nonconvex SF algorithm may tighten to $O(1/T)$. Our work extends SF's horizon-free guarantees to smooth nonconvex optimization and charts future directions for optimal nonconvex rates.