Learning Rate Optimization for Deep Neural Networks Using Lipschitz Bandits

TL;DR

This paper introduces a Lipschitz bandit-based method for tuning neural network learning rates, achieving faster and more efficient training compared to existing hyperparameter optimization techniques.

Contribution

It presents a novel Lipschitz bandit-driven approach for learning rate optimization, outperforming HyperOpt and BLiE in efficiency and effectiveness.

Findings

Fewer evaluations needed to find optimal learning rate

Achieves higher test accuracy with less training time

Reduces computational cost of hyperparameter tuning

Abstract

Learning rate is a crucial parameter in training of neural networks. A properly tuned learning rate leads to faster training and higher test accuracy. In this paper, we propose a Lipschitz bandit-driven approach for tuning the learning rate of neural networks. The proposed approach is compared with the popular HyperOpt technique used extensively for hyperparameter optimization and the recently developed bandit-based algorithm BLiE. The results for multiple neural network architectures indicate that our method finds a better learning rate using a) fewer evaluations and b) lesser number of epochs per evaluation, when compared to both HyperOpt and BLiE. Thus, the proposed approach enables more efficient training of neural networks, leading to lower training time and lesser computational cost.