Dynamic Learning Rate Scheduling based on Loss Changes Leads to Faster Convergence

TL;DR

This paper introduces GreedyLR, a novel adaptive learning rate scheduler that adjusts based on loss changes, leading to faster convergence and improved accuracy across NLP, CV, and LLM tasks.

Contribution

The paper presents GreedyLR, a new adaptive scheduler with theoretical convergence guarantees and practical robustness, outperforming traditional schedulers in various large-scale tasks.

Findings

GreedyLR improves convergence speed and accuracy.

It outperforms cosine and exponential decay schedulers.

Theoretical analysis confirms convergence and robustness.

Abstract

Despite significant advances in optimizers for training, most research works use common scheduler choices like Cosine or exponential decay. In this paper, we study \emph{GreedyLR}, a novel scheduler that adaptively adjusts the learning rate during training based on the current loss. To validate the effectiveness of our proposed scheduler, we conduct experiments on several NLP, CV, and LLM tasks with up to $7B$ parameters, including both fine-tuning and pre-training experiments. The results show that our approach outperforms several state-of-the-art schedulers in terms of accuracy, speed, and convergence. We also provide a theoretical analysis of the GreedyLR algorithm, including a proof of convergence and derivation of the optimal scaling factor $F$ that maximizes the convergence rate, along with experiments to show robustness of the algorithm to realistic noisy landscapes. Our scheduler is easy to implement, computationally efficient, and could be considered a good default scheduler for training.