AutoLR: An Evolutionary Approach to Learning Rate Policies

TL;DR

AutoLR introduces an evolutionary framework that customizes learning rate policies for specific neural network architectures, leading to more efficient training compared to standard baseline methods.

Contribution

It presents a novel method using Structured Grammatical Evolution to evolve tailored learning rate schedulers for individual neural network topologies.

Findings

Evolved policies outperform baseline learning rates in training efficiency.

The approach demonstrates the viability of evolutionary methods for hyperparameter customization.

Customized policies improve neural network performance on specific architectures.

Abstract

The choice of a proper learning rate is paramount for good Artificial Neural Network training and performance. In the past, one had to rely on experience and trial-and-error to find an adequate learning rate. Presently, a plethora of state of the art automatic methods exist that make the search for a good learning rate easier. While these techniques are effective and have yielded good results over the years, they are general solutions. This means the optimization of learning rate for specific network topologies remains largely unexplored. This work presents AutoLR, a framework that evolves Learning Rate Schedulers for a specific Neural Network Architecture using Structured Grammatical Evolution. The system was used to evolve learning rate policies that were compared with a commonly used baseline value for learning rate. Results show that training performed using certain evolved policies is more efficient than the established baseline and suggest that this approach is a viable means of improving a neural network's performance.