Statistics of correlations in nonlinear recurrent neural networks

TL;DR

This paper derives exact formulas for correlation statistics in large nonlinear recurrent neural networks, extending linear results and including nonlinear activation effects with analytical and numerical validation.

Contribution

It introduces a path-integral approach to analyze nonlinear network correlations, generalizing previous linear models and providing explicit results for specific activation functions.

Findings

Exact correlation statistics derived for large nonlinear networks

Power-law activations exhibit scaling behavior controlled by network coupling

Analytic predictions for Pade approximant activation functions are confirmed by simulations

Abstract

The statistics of correlations are central quantities characterizing the collective dynamics of recurrent neural networks. We derive exact expressions for the statistics of correlations of nonlinear recurrent networks in the limit of a large number N of neurons, including systematic 1/N corrections, in the regime of Gaussian quenched disorder. Our approach uses a path-integral representation of the network stochastic dynamics, which reduces the description to a few collective variables and enables efficient computation. This generalizes previous results on linear networks to include a wide family of nonlinear activation functions, which enter as interaction terms in the path integral. These interactions can resolve the instability of the linear theory and yield a strictly positive participation dimension. We present explicit results for power-law activations, revealing scaling behavior controlled by the network coupling. In addition, we introduce a class of activation functions based on Pade approximants and provide analytic predictions for their correlation statistics. Numerical simulations confirm our theoretical results with excellent agreement. We also compare with previous works that have studied the complementary case with annealed disorder, and based on this we propose a new self-consistent equation for the more general case of colored noise.