Meta-stable states in the hierarchical Dyson model drive parallel processing in the hierarchical Hopfield network

TL;DR

This paper explores hierarchical neural networks using statistical mechanics, revealing their capacity for both serial and parallel processing due to meta-stable states, and extends the analysis to multi-pattern Hopfield-like networks with hierarchical topology.

Contribution

It introduces a novel analysis of hierarchical neural networks' stability and processing capabilities, combining mean-field and fluctuation reabsorption techniques, and extends the framework to multi-pattern networks with proven thermodynamic limits.

Findings

Hierarchical Dyson model supports both serial and parallel processing.

Existence of thermodynamic limit for hierarchical Hopfield networks with low storage.

Explicit bounds for free energy in hierarchical neural networks.

Abstract

In this paper we introduce and investigate the statistical mechanics of hierarchical neural networks: First, we approach these systems \`a la Mattis, by thinking at the Dyson model as a single-pattern hierarchical neural network and we discuss the stability of different retrievable states as predicted by the related self-consistencies obtained from a mean-field bound and from a bound that bypasses the mean-field limitation. The latter is worked out by properly reabsorbing fluctuations of the magnetization related to higher levels of the hierarchy into effective fields for the lower levels. Remarkably, mixing Amit's ansatz technique (to select candidate retrievable states) with the interpolation procedure (to solve for the free energy of these states) we prove that (due to gauge symmetry) the Dyson model accomplishes both serial and parallel processing. One step forward, we extend this scenario toward multiple stored patterns by implementing the Hebb prescription for learning within the couplings. This results in an Hopfield-like networks constrained on a hierarchical topology, for which, restricting to the low storage regime (where the number of patterns grows at most logarithmical with the amount of neurons), we prove the existence of the thermodynamic limit for the free energy and we give an explicit expression of its mean field bound and of the related improved bound