AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop

TL;DR

AutoDrop introduces an automatic learning rate drop method based on the angular velocity of model parameters, improving training speed and generalization without extra hyperparameters.

Contribution

The paper proposes AutoDrop, a novel algorithm that automatically determines when to drop the learning rate based on angular velocity, eliminating manual scheduling and hyperparameter tuning.

Findings

Accelerates deep learning training processes.

Achieves better generalization performance.

Does not require additional hyperparameter tuning.

Abstract

Modern deep learning (DL) architectures are trained using variants of the SGD algorithm that is run with a $\textit{manually}$ defined learning rate schedule, i.e., the learning rate is dropped at the pre-defined epochs, typically when the training loss is expected to saturate. In this paper we develop an algorithm that realizes the learning rate drop $\textit{automatically}$. The proposed method, that we refer to as AutoDrop, is motivated by the observation that the angular velocity of the model parameters, i.e., the velocity of the changes of the convergence direction, for a fixed learning rate initially increases rapidly and then progresses towards soft saturation. At saturation the optimizer slows down thus the angular velocity saturation is a good indicator for dropping the learning rate. After the drop, the angular velocity "resets" and follows the previously described pattern - it increases again until saturation. We show that our method improves over SOTA training approaches: it accelerates the training of DL models and leads to a better generalization. We also show that our method does not require any extra hyperparameter tuning. AutoDrop is furthermore extremely simple to implement and computationally cheap. Finally, we develop a theoretical framework for analyzing our algorithm and provide convergence guarantees.