Covariance spectrum in nonlinear recurrent neural networks

TL;DR

This paper extends the theoretical understanding of the covariance spectrum in neural networks to include nonlinear neurons and chaotic regimes, showing that the spectrum can be accurately predicted by an effective connection strength parameter.

Contribution

It introduces a nonlinear theory for the covariance spectrum in recurrent neural networks, bridging the gap between minimal linear models and biological neural dynamics.

Findings

The spectrum can be understood using an effective connection strength parameter.

The effective connection strength remains near critical in chaotic regimes.

The theory applies across various dynamical regimes, including chaos.

Abstract

Advances in simultaneous recordings of large numbers of neurons have driven significant interest in the structure of neural population activity such as dimension. A key question is how these dynamic features arise mechanistically and their relationship to circuit connectivity. It was previously proposed to use the covariance eigenvalue distribution, or spectrum, which can be analytically derived in random recurrent networks, as a robust measure to describe the shape of neural population activity beyond the dimension (Hu and Sompolinsky 2022). Applications of the theoretical spectrum have broadly found accurate matches to experimental data across brain areas providing mechanistic insights into the observed low dimensional population dynamics (Morales et al. 2023). However, the empirical success highlights a gap in theory, as the neural network model used to derive the spectrum was minimal with linear neurons. In this work, we aim to close this gap by studying the covariance spectrum in networks with nonlinear neurons and under broader dynamical regimes including chaos. Surprisingly, we found that the spectrum can be precisely understood by equations analogous to the linear theory substituted with an effective recurrent connection strength parameter, that reflects both the connection weights and the nonlinearity of neurons. Across dynamical regimes, this effective connection strength provides a unified interpretation for the spectrum and dimension changes, and stays near the critical value in the chaotic regime without fine-tuning. These results further our understanding of nonlinear neural population dynamics and provide additional theoretical support for applying the covariance spectrum analysis in biological circuits.