Stochastic Thermodynamics of Associative Memory

TL;DR

This paper explores the thermodynamic costs and out-of-equilibrium properties of Dense Associative Memory networks, linking their computational dynamics to energy landscapes and entropy production.

Contribution

It introduces a framework for analyzing thermodynamic entropy production in DenseAMs and characterizes their out-of-equilibrium behavior using dynamical mean field theory.

Findings

Characterized entropy production during memory retrieval.

Identified a failure mode in higher order networks at non-zero temperature.

Developed a method to calculate work and power costs in the mean field limit.

Abstract

Dense Associative Memory networks (DenseAMs) unify several popular paradigms in Artificial Intelligence (AI), such as Hopfield Networks, transformers, and diffusion models, while casting their computational properties into the language of dynamical systems and energy landscapes. This formulation provides a natural setting for studying thermodynamics and computation in neural systems, because DenseAMs are simultaneously simple enough to admit analytic treatment and rich enough to implement nontrivial computational function. Aspects of these networks have been studied at equilibrium and at zero temperature, but the thermodynamic costs associated with their operation out of equilibrium are largely unexplored. Here, we define the thermodynamic entropy production associated with the operation of such networks, and study polynomial DenseAMs at intermediate memory load. At large system sizes and intermediate and low load, we use dynamical mean field theory to characterize out-of-equilibrium properties, work requirements, and memory transition times when driving the system with corrupted memories. We characterize a failure mode of higher order networks not observed at zero temperature. Further, we develop a method for calculating work and power costs in the mean field limit. Finally, we find tradeoffs between entropy production, memory retrieval accuracy, and operation speed.