REX: Revisiting Budgeted Training with an Improved Schedule

TL;DR

This paper introduces the REX schedule, a novel learning rate schedule that adapts to different training budgets, outperforming traditional schedules in low budget scenarios and matching or exceeding them in high budget cases without extra costs.

Contribution

The paper proposes the REX schedule, a new adaptive learning rate profile and sampling rate combination that improves budget-aware training performance.

Findings

REX outperforms linear schedule in low budget regimes.

REX matches or exceeds state-of-the-art schedules in various settings.

REX requires no additional computation, storage, or hyperparameters.

Abstract

Deep learning practitioners often operate on a computational and monetary budget. Thus, it is critical to design optimization algorithms that perform well under any budget. The linear learning rate schedule is considered the best budget-aware schedule, as it outperforms most other schedules in the low budget regime. On the other hand, learning rate schedules -- such as the \texttt{30-60-90} step schedule -- are known to achieve high performance when the model can be trained for many epochs. Yet, it is often not known a priori whether one's budget will be large or small; thus, the optimal choice of learning rate schedule is made on a case-by-case basis. In this paper, we frame the learning rate schedule selection problem as a combination of $i)$ selecting a profile (i.e., the continuous function that models the learning rate schedule), and $ii)$ choosing a sampling rate (i.e., how frequently the learning rate is updated/sampled from this profile). We propose a novel profile and sampling rate combination called the Reflected Exponential (REX) schedule, which we evaluate across seven different experimental settings with both SGD and Adam optimizers. REX outperforms the linear schedule in the low budget regime, while matching or exceeding the performance of several state-of-the-art learning rate schedules (linear, step, exponential, cosine, step decay on plateau, and OneCycle) in both high and low budget regimes. Furthermore, REX requires no added computation, storage, or hyperparameters.