Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates

TL;DR

This paper introduces a cyclic learning rate schedule that efficiently constructs accuracy versus training time tradeoff curves in a single run, enabling rapid evaluation of training methods.

Contribution

The paper presents a novel cyclic learning rate approach to quickly generate tradeoff curves, reducing the computational cost of benchmarking neural network training efficiency.

Findings

Cyclic learning rates produce reliable tradeoff curves in one training run.

The method effectively compares different training techniques like Blurpool and MixUp.

Tradeoff curves reveal the impact of algorithmic choices on training efficiency.

Abstract

Benchmarking the tradeoff between neural network accuracy and training time is computationally expensive. Here we show how a multiplicative cyclic learning rate schedule can be used to construct a tradeoff curve in a single training run. We generate cyclic tradeoff curves for combinations of training methods such as Blurpool, Channels Last, Label Smoothing and MixUp, and highlight how these cyclic tradeoff curves can be used to evaluate the effects of algorithmic choices on network training efficiency.