Understanding Transformer from the Perspective of Associative Memory

TL;DR

This paper offers a new perspective on Transformer architectures by analyzing them as associative memory systems, providing insights into their capacity, learning dynamics, and potential for future improvements.

Contribution

It introduces a framework connecting Transformers with associative memory concepts, including measures like retrieval SNR and insights into memory capacity and update mechanisms.

Findings

Softmax Attention's effectiveness explained via kernel perspective

FFNs can be viewed as associative memory units

Transformers' limitations and potential for infinite context explored

Abstract

In this paper, we share our reflections and insights on understanding Transformer architectures through the lens of associative memory--a classic psychological concept inspired by human cognition. We start with the basics of associative memory (think simple linear attention) and then dive into two dimensions: Memory Capacity: How much can a Transformer really remember, and how well? We introduce retrieval SNR to measure this and use a kernel perspective to mathematically reveal why Softmax Attention is so effective. We also show how FFNs can be seen as a type of associative memory, leading to insights on their design and potential improvements. Memory Update: How do these memories learn and evolve? We present a unified framework for understanding how different Transformer variants (like DeltaNet and Softmax Attention) update their "knowledge base". This leads us to tackle two provocative questions: 1. Are Transformers fundamentally limited in what they can express, and can we break these barriers? 2. If a Transformer had infinite context, would it become infinitely intelligent? We want to demystify Transformer architecture, offering a clearer understanding of existing designs. This exploration aims to provide fresh insights and spark new avenues for Transformer innovation.