Dynamical Properties of Dense Associative Memory

TL;DR

This paper analyzes the dynamical behavior of dense associative memory, a modern Hopfield network variant, using generating functional analysis to evaluate convergence, capacity, and robustness of pattern retrieval.

Contribution

It introduces an exact analytical approach to study the dynamics of dense associative memory, revealing insights into convergence, capacity, and robustness not previously explored.

Findings

Convergence time and attraction basin sizes are quantitatively characterized.

Retrieval in dense associative memory is more robust than in traditional Hopfield models.

The methodology can be applied to other energy-based neural network models.

Abstract

Dense associative memory, a fundamental instance of modern Hopfield networks, can store a large number of memory patterns as equilibrium states of recurrent networks. While the stationary-state storage capacity has been investigated, its dynamical properties have not yet been discussed. In this paper, we analyze the dynamics using an exact approach based on generating functional analysis. We show results on convergence properties of memory retrieval, such as the convergence time and the size of the attraction basins. Our analysis enables a quantitative evaluation of the convergence time and the storage capacity of dense associative memory, which is useful for model design. Unlike the traditional Hopfield model, the retrieval of a pattern does not act as additional noise to itself, suggesting that the structure of modern networks makes recall more robust. Furthermore, the methodology addressed here can be applied to other energy-based models, and thus has the potential to contribute to the design of future architectures.