Using a one dimensional parabolic model of the full-batch loss to estimate learning rates during training

TL;DR

This paper proposes a novel line search method for deep learning that uses a one-dimensional parabolic model of the full-batch loss to adaptively estimate learning rates during training, addressing the challenge of automatic step size selection.

Contribution

It introduces a parabolic approximation-based line search method that efficiently estimates learning rates using mini-batches, outperforming existing approaches in various deep learning scenarios.

Findings

Method matches tuned SGD with Momentum in performance.

Often outperforms other line search methods across models and datasets.

First to sample larger batch sizes over multiple inferences in deep learning line search.

Abstract

A fundamental challenge in Deep Learning is to find optimal step sizes for stochastic gradient descent automatically. In traditional optimization, line searches are a commonly used method to determine step sizes. One problem in Deep Learning is that finding appropriate step sizes on the full-batch loss is unfeasibly expensive. Therefore, classical line search approaches, designed for losses without inherent noise, are usually not applicable. Recent empirical findings suggest, inter alia, that the full-batch loss behaves locally parabolically in the direction of noisy update step directions. Furthermore, the trend of the optimal update step size changes slowly. By exploiting these and more findings, this work introduces a line-search method that approximates the full-batch loss with a parabola estimated over several mini-batches. Learning rates are derived from such parabolas during training. In the experiments conducted, our approach is on par with SGD with Momentum tuned with a piece-wise constant learning rate schedule and often outperforms other line search approaches for Deep Learning across models, datasets, and batch sizes on validation and test accuracy. In addition, our approach is the first line search approach for Deep Learning that samples a larger batch size over multiple inferences to still work in low-batch scenarios.