Theory of Recurrent Neural Network with Common Synaptic Inputs

TL;DR

This paper develops a theoretical framework to analyze the effects of common synaptic inputs in recurrent neural networks, addressing the challenge of sample dependence and lack of self-averaging.

Contribution

It introduces a macroscopic dynamical description for recurrent neural networks with common inputs, extending analysis beyond layered networks.

Findings

Derived a recurrence relation for the probability density function of the network state.

Showed that common synaptic inputs induce sample dependence and break self-averaging.

Provided a new theoretical approach for analyzing recurrent neural networks with feedback.

Abstract

We discuss the effects of common synaptic inputs in a recurrent neural network. Because of the effects of these common synaptic inputs, the correlation between neural inputs cannot be ignored, and thus the network exhibits sample dependence. Networks of this type do not have well-defined thermodynamic limits, and self-averaging breaks down. We therefore need to develop a suitable theory without relying on these common properties. While the effects of the common synaptic inputs have been analyzed in layered neural networks, it was apparently difficult to analyze these effects in recurrent neural networks due to feedback connections. We investigated a sequential associative memory model as an example of recurrent networks and succeeded in deriving a macroscopic dynamical description as a recurrence relation form of a probability density function.