Statistical mechanics of vector Hopfield network near and above saturation

TL;DR

This paper analyzes the equilibrium and out-of-equilibrium properties of vector Hopfield networks, revealing how increasing spin dimension affects storage capacity, denoising ability, and the energy landscape's spectral properties.

Contribution

It derives the replica symmetric solution for vector Hopfield networks, characterizes their phase diagram, and explores how capacity and denoising capabilities scale with spin dimension.

Findings

Retrieval phase shrinks as spin dimension increases, with critical capacity decreasing as 1/d.

Vector Hopfield networks can denoise inputs effectively up to capacities proportional to d.

Energy landscape minima have soft, localized eigenmodes related to noisy neurons.

Abstract

We study analytically and numerically a Hopfield fully-connected network with $d$-dimensional vector spins. These networks are models of associative memory that generalize the standard Hopfield with Ising spins, where $P$ examples are stored in a network of $N$ units as local minima in an energy landscape. We study the equilibrium and out-of-equilibrium properties of the system, considering the system in its retrieval phase $\alpha<\alpha_c$ and beyond, where $\alpha=P/N$ is the capacity of the system and $\alpha_c$ is its critical value, above which storage fails. We derive the Replica Symmetric solution for the equilibrium thermodynamics of the system, together with its phase diagram: we find that the retrieval phase of the network shrinks with growing spin dimension, having ultimately a vanishing critical capacity $\alpha_c\propto 1/d$ in the large $d$ limit. As a trade-off, we observe that in the same limit vector Hopfield networks are able to denoise corrupted input patterns in the first step of retrieval dynamics, up to very large capacities $\alpha\propto d$. We also study the static properties of the system at zero temperature, considering the statistical properties of soft modes of the energy Hessian spectrum. We find that local minima of the energy landscape related to memory states have ungapped spectra with rare soft eigenmodes: these excitations are localized, their measure condensating on the noisiest neurons of the memory state.