Bridging Associative Memory and Probabilistic Modeling

TL;DR

This paper establishes a conceptual link between associative memory and probabilistic modeling, introducing new models and analyses that enhance understanding and capabilities in both areas.

Contribution

It introduces a theoretical framework connecting associative memory energy functions with probabilistic models, along with novel models and analytical tools.

Findings

New energy-based models for in-context learning of energy functions

Two associative memory models using Bayesian nonparametrics and evidence lower bound

Characterization of Gaussian kernel density estimators' memory capacity

Abstract

Associative memory and probabilistic modeling are two fundamental topics in artificial intelligence. The first studies recurrent neural networks designed to denoise, complete and retrieve data, whereas the second studies learning and sampling from probability distributions. Based on the observation that associative memory's energy functions can be seen as probabilistic modeling's negative log likelihoods, we build a bridge between the two that enables useful flow of ideas in both directions. We showcase four examples: First, we propose new energy-based models that flexibly adapt their energy functions to new in-context datasets, an approach we term \textit{in-context learning of energy functions}. Second, we propose two new associative memory models: one that dynamically creates new memories as necessitated by the training data using Bayesian nonparametrics, and another that explicitly computes proportional memory assignments using the evidence lower bound. Third, using tools from associative memory, we analytically and numerically characterize the memory capacity of Gaussian kernel density estimators, a widespread tool in probababilistic modeling. Fourth, we study a widespread implementation choice in transformers -- normalization followed by self attention -- to show it performs clustering on the hypersphere. Altogether, this work urges further exchange of useful ideas between these two continents of artificial intelligence.