An Adaptive Volatility-based Learning Rate Scheduler

TL;DR

This paper introduces VolSched, an adaptive learning rate scheduler inspired by volatility in stochastic processes, which improves training efficiency and generalization in deep neural networks by dynamically adjusting the learning rate based on accuracy volatility.

Contribution

The paper presents VolSched, a novel adaptive LR scheduler that uses accuracy volatility ratios to enhance exploration and generalization in neural network training.

Findings

VolSched improves top-1 accuracy by over 1.3 percentage points on CIFAR-100.

Models trained with VolSched find flatter minima, indicating better generalization.

VolSched promotes longer exploration phases during training.

Abstract

Effective learning rate (LR) scheduling is crucial for training deep neural networks. However, popular pre-defined and adaptive schedulers can still lead to suboptimal generalization. This paper introduces VolSched, a novel adaptive LR scheduler inspired by the concept of volatility in stochastic processes like Geometric Brownian Motion to dynamically adjust the learning rate. By calculating the ratio between long-term and short-term accuracy volatility, VolSched increases the LR to escape plateaus and decreases it to stabilize training, allowing the model to explore the loss landscape more effectively. We evaluate VolSched on the CIFAR-100 dataset against a strong baseline using a standard augmentation pipeline. When paired with ResNet-18 and ResNet-34, our scheduler delivers consistent performance gains, improving top-1 accuracy by 1.4 and 1.3 percentage points respectively. Analysis of the loss curves reveals that VolSched promotes a longer exploration phase. A quantitative analysis of the Hessian shows that VolSched finds a final solution that is 38% flatter than the next-best baseline, allowing the model to obtain wider minima and hence better generalization performance.