Leveraging chaotic transients in the training of artificial neural networks

TL;DR

This paper investigates how large learning rates induce chaotic dynamics during neural network training, revealing that transient chaos can accelerate learning and improve training efficiency across various architectures and tasks.

Contribution

It demonstrates that chaotic transients at high learning rates can be harnessed to optimize neural network training, a novel insight into the role of chaos in learning dynamics.

Findings

Optimal training speed occurs near the onset of chaos.

Chaotic dynamics are observed across different architectures and tasks.

Transient chaos can be beneficial for faster convergence.

Abstract

Traditional algorithms to optimize artificial neural networks when confronted with a supervised learning task are usually exploitation-type relaxational dynamics such as gradient descent (GD). Here, we explore the dynamics of the neural network trajectory along training for unconventionally large learning rates. We show that for a region of values of the learning rate, the GD optimization shifts away from purely exploitation-like algorithm into a regime of exploration-exploitation balance, as the neural network is still capable of learning but the trajectory shows sensitive dependence on initial conditions --as characterized by positive network maximum Lyapunov exponent--. Interestingly, the characteristic training time required to reach an acceptable accuracy in the test set reaches a minimum precisely in such learning rate region, further suggesting that one can accelerate the training of artificial neural networks by locating at the onset of chaos. Our results --initially illustrated for the MNIST classification task-- qualitatively hold for a range of supervised learning tasks, {learning architectures (including both shallow and deep multilayer perceptrons and convolutional neural networks) and other hyperparameters (different activation functions and weight regularisation),} and showcase the emergent, constructive role of transient chaotic dynamics in the training of artificial neural networks.