Modern Methods in Associative Memory

TL;DR

This paper reviews modern associative memory models, highlighting their theoretical advancements, connections to current AI architectures, and practical design methods, with a focus on their potential for understanding and developing neural network systems.

Contribution

It provides an accessible overview of recent developments in associative memories, including new theoretical insights, Lagrangian formulations, and their relevance to modern AI architectures.

Findings

Recent theoretical results enhance understanding of storage capacities.

Connections established between associative memories and SOTA AI models.

New formulations enable design of distributed neural architectures.

Abstract

Associative Memories like the famous Hopfield Networks are elegant models for describing fully recurrent neural networks whose fundamental job is to store and retrieve information. In the past few years they experienced a surge of interest due to novel theoretical results pertaining to their information storage capabilities, and their relationship with SOTA AI architectures, such as Transformers and Diffusion Models. These connections open up possibilities for interpreting the computation of traditional AI networks through the theoretical lens of Associative Memories. Additionally, novel Lagrangian formulations of these networks make it possible to design powerful distributed models that learn useful representations and inform the design of novel architectures. This tutorial provides an approachable introduction to Associative Memories, emphasizing the modern language and methods used in this area of research, with practical hands-on mathematical derivations and coding notebooks.