Lyapunov Learning at the Onset of Chaos

TL;DR

Lyapunov Learning is a novel training method that prepares neural networks for regime shifts by leveraging chaos theory, enabling rapid adaptation in non-stationary systems with significant performance improvements.

Contribution

The paper introduces Lyapunov Learning, a new training algorithm that uses properties of chaotic dynamical systems to enhance neural network adaptability to regime shifts.

Findings

Outperforms regular training in non-stationary systems

Achieves 96% increase in loss ratio during regime shifts

Effective in adapting to abrupt changes in Lorenz system parameters

Abstract

Handling regime shifts and non-stationary time series in deep learning systems presents a significant challenge. In the case of online learning, when new information is introduced, it can disrupt previously stored data and alter the model's overall paradigm, especially with non-stationary data sources. Therefore, it is crucial for neural systems to quickly adapt to new paradigms while preserving essential past knowledge relevant to the overall problem. In this paper, we propose a novel training algorithm for neural networks called \textit{Lyapunov Learning}. This approach leverages the properties of nonlinear chaotic dynamical systems to prepare the model for potential regime shifts. Drawing inspiration from Stuart Kauffman's Adjacent Possible theory, we leverage local unexplored regions of the solution space to enable flexible adaptation. The neural network is designed to operate at the edge of chaos, where the maximum Lyapunov exponent, indicative of a system's sensitivity to small perturbations, evolves around zero over time. Our approach demonstrates effective and significant improvements in experiments involving regime shifts in non-stationary systems. In particular, we train a neural network to deal with an abrupt change in Lorenz's chaotic system parameters. The neural network equipped with Lyapunov learning significantly outperforms the regular training, increasing the loss ratio by about $96\%$.