Self-attention as an attractor network: transient memories without backpropagation

TL;DR

This paper interprets self-attention in transformers as an attractor network derived from local energy terms, enabling training without backpropagation and revealing transient memory states linked to data.

Contribution

It introduces a novel framework to view self-attention as an attractor network based on energy functions, allowing backpropagation-free training and new theoretical insights.

Findings

Self-attention can be derived from local energy terms.

The proposed model exhibits transient states correlated with data.

Training without backpropagation is feasible using this framework.

Abstract

Transformers are one of the most successful architectures of modern neural networks. At their core there is the so-called attention mechanism, which recently interested the physics community as it can be written as the derivative of an energy function in certain cases: while it is possible to write the cross-attention layer as a modern Hopfield network, the same is not possible for the self-attention, which is used in the GPT architectures and other autoregressive models. In this work we show that it is possible to obtain the self-attention layer as the derivative of local energy terms, which resemble a pseudo-likelihood. We leverage the analogy with pseudo-likelihood to design a recurrent model that can be trained without backpropagation: the dynamics shows transient states that are strongly correlated with both train and test examples. Overall we present a novel framework to interpret self-attention as an attractor network, potentially paving the way for new theoretical approaches inspired from physics to understand transformers.