The Road Less Scheduled

TL;DR

This paper introduces a schedule-free learning rate approach that achieves state-of-the-art performance across various problems without requiring a predefined stopping time or additional hyperparameters.

Contribution

It presents a novel schedule-free method based on a new theory unifying scheduling and iterate averaging, eliminating the need for tuning schedules.

Findings

Outperforms schedule-dependent methods across diverse problems

No additional hyperparameters needed over standard optimizers

Achieved top results in MLCommons 2024 challenge

Abstract

Existing learning rate schedules that do not require specification of the optimization stopping step T are greatly out-performed by learning rate schedules that depend on T. We propose an approach that avoids the need for this stopping time by eschewing the use of schedules entirely, while exhibiting state-of-the-art performance compared to schedules across a wide family of problems ranging from convex problems to large-scale deep learning problems. Our Schedule-Free approach introduces no additional hyper-parameters over standard optimizers with momentum. Our method is a direct consequence of a new theory we develop that unifies scheduling and iterate averaging. An open source implementation of our method is available at https://github.com/facebookresearch/schedule_free. Schedule-Free AdamW is the core algorithm behind our winning entry to the MLCommons 2024 AlgoPerf Algorithmic Efficiency Challenge Self-Tuning track.