The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training

TL;DR

This paper reveals that learning-rate schedules for large models align closely with convex optimization bounds, enabling improved tuning and training efficiency by leveraging theoretical insights.

Contribution

It demonstrates the practical relevance of convex optimization theory to learning-rate scheduling and introduces methods for better tuning based on this understanding.

Findings

Learning-rate schedules match convex optimization bounds closely.

Extending schedules with optimal learning-rate improves training.

Transferring optimal learning-rate across schedules enhances performance.

Abstract

We show that learning-rate schedules for large model training behave surprisingly similar to a performance bound from non-smooth convex optimization theory. We provide a bound for the constant schedule with linear cooldown; in particular, the practical benefit of cooldown is reflected in the bound due to the absence of logarithmic terms. Further, we show that this surprisingly close match between optimization theory and practice can be exploited for learning-rate tuning: we achieve noticeable improvements for training 124M and 210M Llama-type models by (i) extending the schedule for continued training with optimal learning-rate, and (ii) transferring the optimal learning-rate across schedules.