Phase Diagram and Storage Capacity of Sequence Processing Neural Networks

TL;DR

This paper analyzes the dynamics of sequence-storing neural networks, deriving exact equations and revealing a larger storage capacity compared to static pattern networks, validated by simulations.

Contribution

It provides an exact dynamical analysis of asymmetric sequence neural networks and demonstrates an increased storage capacity over static pattern models.

Findings

Derived exact dynamical equations for sequence neural networks.

Identified a larger storage capacity of approximately 0.269.

Validated theoretical results with extensive computer simulations.

Abstract

We solve the dynamics of Hopfield-type neural networks which store sequences of patterns, close to saturation. The asymmetry of the interaction matrix in such models leads to violation of detailed balance, ruling out an equilibrium statistical mechanical analysis. Using generating functional methods we derive exact closed equations for dynamical order parameters, viz. the sequence overlap and correlation- and response functions, in the thermodynamic limit. We calculate the time translation invariant solutions of these equations, describing stationary limit-cycles, which leads to a phase diagram. The effective retarded self-interaction usually appearing in symmetric models is here found to vanish, which causes a significantly enlarged storage capacity of $\alpha_c\sim 0.269$, compared to $\alpha_\c\sim 0.139$ for Hopfield networks storing static patterns. Our results are tested against extensive computer simulations and excellent agreement is found.