ExpTest: Automating Learning Rate Searching and Tuning with Insights from Linearized Neural Networks

TL;DR

ExpTest is a novel method that automates initial learning rate search and tuning for deep neural networks by leveraging insights from linearized neural networks and real-time loss curve analysis, reducing manual effort and resource costs.

Contribution

It introduces ExpTest, a new approach that eliminates the need for manual initial learning rate selection and scheduling, with theoretical justification and empirical validation.

Findings

Achieves state-of-the-art performance across various tasks and architectures.

Requires minimal hyperparameter tuning and overhead.

Robust to hyperparameter choices and does not need initial learning rate setting.

Abstract

Hyperparameter tuning remains a significant challenge for the training of deep neural networks (DNNs), requiring manual and/or time-intensive grid searches, increasing resource costs and presenting a barrier to the democratization of machine learning. The global initial learning rate for DNN training is particularly important. Several techniques have been proposed for automated learning rate tuning during training; however, they still require manual searching for the global initial learning rate. Though methods exist that do not require this initial selection, they suffer from poor performance. Here, we present ExpTest, a sophisticated method for initial learning rate searching and subsequent learning rate tuning for the training of DNNs. ExpTest draws on insights from linearized neural networks and the form of the loss curve, which we treat as a real-time signal upon which we perform hypothesis testing. We mathematically justify ExpTest and provide empirical support. ExpTest requires minimal overhead, is robust to hyperparameter choice, and achieves state-of-the-art performance on a variety of tasks and architectures, without initial learning rate selection or learning rate scheduling.