Context-Gated Associative Retrieval: From Theory to Transformers

TL;DR

This paper introduces a context-gated associative memory model that enhances retrieval by incorporating external context, with theoretical and empirical validation linking it to transformer-based in-context learning.

Contribution

It proposes a novel two-stage memory architecture with context gating, providing theoretical insights and demonstrating its relevance to transformer models like Llama-3.

Findings

Context gating increases memory separation and sparsity.

The system admits a unique fixed point driven by context and feedback.

In-context learning in transformers acts as context-gated retrieval.

Abstract

Hopfield networks and their generalizations have established deep connections among biological associative memories, statistical physics, and transformers. Yet most models treat retrieval as a fixed query-to-memory mapping, ignoring the role of external context in recall. In this work, we propose a two-stage associative memory architecture, wherein a context-gate subcircuit reshapes the retrieval energy landscape before and during recall. We show theoretically that context gating increases inter-memory separation while inducing sparsity, translating into exponential improvements in retrieval. Crucially, we prove that the system admits a unique self-consistent fixed point, revealing that the resulting retrieval state is driven by both a direct contextual bias and a second-order retrieval-gate feedback loop. We then bridge this theory to transformers; specifically, we evaluate a first-order approximation on Llama-3, confirming that in-context learning acts as context-gated retrieval. Native dynamics mirror our theory: context localizes a memory subspace, enabling the zero-shot query to cleanly discriminate. Ultimately, this framework provides a mechanistic link between associative memory theory and LLM phenomenology.