HyperbolicLR: Epoch insensitive learning rate scheduler

TL;DR

This paper introduces two hyperbolic-based learning rate schedulers that provide epoch-insensitive, stable, and consistent training performance across various deep learning tasks, addressing a common issue with traditional schedulers.

Contribution

The paper presents novel hyperbolic and exponential hyperbolic learning rate schedulers that maintain stable learning curves regardless of epoch count, improving robustness in deep network training.

Findings

Achieve more consistent performance across different epoch settings

Demonstrate improved stability in training curves on multiple tasks

Offer a more robust alternative to conventional learning rate schedulers

Abstract

This study proposes two novel learning rate schedulers -- Hyperbolic Learning Rate Scheduler (HyperbolicLR) and Exponential Hyperbolic Learning Rate Scheduler (ExpHyperbolicLR) -- to address the epoch sensitivity problem that often causes inconsistent learning curves in conventional methods. By leveraging the asymptotic behavior of hyperbolic curves, the proposed schedulers maintain more stable learning curves across varying epoch settings. Specifically, HyperbolicLR applies this property directly in the epoch-learning rate space, while ExpHyperbolicLR extends it to an exponential space. We first determine optimal hyperparameters for each scheduler on a small number of epochs, fix these hyperparameters, and then evaluate performance as the number of epochs increases. Experimental results on various deep learning tasks (e.g., image classification, time series forecasting, and operator learning) demonstrate that both HyperbolicLR and ExpHyperbolicLR achieve more consistent performance improvements than conventional schedulers as training duration grows. These findings suggest that our hyperbolic-based schedulers offer a more robust and efficient approach to deep network optimization, particularly in scenarios constrained by computational resources or time.