On the dimension of pullback attractors in recurrent neural networks

TL;DR

This paper establishes an upper bound on the complexity of the attractors in recurrent neural networks used for modeling chaotic systems, explaining their effectiveness in capturing low-dimensional dynamics despite high-dimensional state spaces.

Contribution

It provides a theoretical upper bound on the box-counting dimension of the reservoir's pullback attractor, linking it to the input system's dimension.

Findings

The box-counting dimension of the pullback attractor is bounded by that of the input space.

Reservoirs exhibit effective low-dimensional dynamics despite high-dimensional state spaces.

Results explain reservoir computers' success in reconstructing attractors and computing invariants.

Abstract

Recurrent neural networks trained via the reservoir computing paradigm have demonstrated remarkable success in learning and reconstructing attractors from chaotic systems, often replicating quantities such as Lyapunov exponents and fractal dimensions. It has recently been conjectured that this is because the reservoir computer embeds the dynamics of the chaotic system in its state space before learning. This conjecture has been established for reservoir computers with linear activation functions and remains open for more general reservoir systems. In this work, we employ a non-autonomous dynamical systems approach to establish an upper bound for the box-counting dimension of the pullback attractor, a subset of the reservoir state space that is approximated during training and prediction phases. We prove that the box-counting dimension of the pullback attractor is bounded above by the box-counting dimension of the space of input sequences with respect to the product topology. In particular, for input sequences originating from an Nin-dimensional smooth dynamical system or their generic continuously differentiable observations, the box-counting dimension of the pullback attractor is bounded above by Nin. The results obtained here highlight the fact that, while a reservoir computer may possess a very high-dimensional state space, it exhibits effective low-dimensional dynamics. Our findings also partly explain why reservoir computers are successful in tasks such as attractor reconstruction and the computation of dynamic invariants like Lyapunov exponents and fractal dimensions.