How glassy are neural networks?

TL;DR

This paper investigates the high storage regime of neural networks with Gaussian patterns, using an exact mapping to bipartite spin glasses to analyze thermodynamic properties and ergodicity breaking.

Contribution

It introduces a novel exact mapping between neural networks and bipartite spin glasses, enabling complete analysis of the annealed region and extension to the broken ergodicity phase.

Findings

Complete control of the annealed region in the high storage regime.

Identification of the critical line and fluctuation behaviors.

Representation of the quenched free energy as a combination of spin-glass free energies.

Abstract

In this paper we continue our investigation on the high storage regime of a neural network with Gaussian patterns. Through an exact mapping between its partition function and one of a bipartite spin glass (whose parties consist of Ising and Gaussian spins respectively), we give a complete control of the whole annealed region. The strategy explored is based on an interpolation between the bipartite system and two independent spin glasses built respectively by dichotomic and Gaussian spins: Critical line, behavior of the principal thermodynamic observables and their fluctuations as well as overlap fluctuations are obtained and discussed. Then, we move further, extending such an equivalence beyond the critical line, to explore the broken ergodicity phase under the assumption of replica symmetry and we show that the quenched free energy of this (analogical) Hopfield model can be described as a linear combination of the two quenched spin-glass free energies even in the replica symmetric framework.