Kinetic energy in random recurrent neural networks

TL;DR

This paper explores how kinetic energy behaves in random recurrent neural networks, revealing a continuous transition at a critical point and linking it to chaotic dynamics, with implications for understanding neural computation.

Contribution

It introduces a kinetic-energy-focused analysis combining theory and simulations, revealing a cubic scaling law at the chaos onset and connecting energy dynamics to network behavior.

Findings

Kinetic energy shifts from zero to positive at a critical coupling.

The transition exhibits cubic scaling near the critical point.

The activity distribution relates to the network's chaotic and gradient dynamics.

Abstract

High-dimensional chaotic dynamics can emerge in a large random recurrent neural network when the synaptic gain crosses a threshold. Recent works showed that the kinetic energy of neural activity links the chaotic dynamics and the supporting unstable fixed points (equilibria) in the phase space. Here, we investigate the kinetic-energy-centric properties of random recurrent neural networks by combining dynamical mean-field theory with extensive numerical simulations. We find that the average kinetic energy shifts continuously from zero to a positive value at a critical value of coupling variance (synaptic gain) and exhibits a cubic scaling behavior near the critical point from above. This scaling behavior is supported by numerical simulations and provides a quantitative characterization of how fast the dynamics change during the onset of chaos. The steady-state activity distribution is further calculated by the theory and compared with simulations on finite-size systems from the kinetic-energy optimization perspective as well. The activity distribution is also analyzed in a geometric angle, establishing a relationship between the original chaotic dynamics and the gradient dynamics of the kinetic energy. The trajectory length on the chaotic manifold can be derived from the stationary kinetic energy, and the associated stationary behavior is analyzed as well. This study provides a kinetic-energy-centric route toward understanding the dynamics landscape of recurrent neural networks, which may provide insights for reservoir computing and even for internal synaptic learning.