Selecting and Composing Learning Rate Policies for Deep Neural Networks

TL;DR

This paper introduces a systematic approach and a recommendation system for selecting and composing effective learning rate policies in deep neural network training, significantly improving accuracy and reducing training time.

Contribution

It presents an LR tuning mechanism, a recommendation system (LRBench), and extends support to different optimizers, enabling better LR policy choices for DNN training.

Findings

Outperforms default LR policies in accuracy and training time.

Reduces training time by 1.6 to 6.7 times.

Effective across various datasets and DNN models.

Abstract

The choice of learning rate (LR) functions and policies has evolved from a simple fixed LR to the decaying LR and the cyclic LR, aiming to improve the accuracy and reduce the training time of Deep Neural Networks (DNNs). This paper presents a systematic approach to selecting and composing an LR policy for effective DNN training to meet desired target accuracy and reduce training time within the pre-defined training iterations. It makes three original contributions. First, we develop an LR tuning mechanism for auto-verification of a given LR policy with respect to the desired accuracy goal under the pre-defined training time constraint. Second, we develop an LR policy recommendation system (LRBench) to select and compose good LR policies from the same and/or different LR functions through dynamic tuning, and avoid bad choices, for a given learning task, DNN model and dataset. Third, we extend LRBench by supporting different DNN optimizers and show the significant mutual impact of different LR policies and different optimizers. Evaluated using popular benchmark datasets and different DNN models (LeNet, CNN3, ResNet), we show that our approach can effectively deliver high DNN test accuracy, outperform the existing recommended default LR policies, and reduce the DNN training time by 1.6$\sim$6.7$\times$ to meet a targeted model accuracy.