Self-Attention Limits Working Memory Capacity of Transformer-Based Models

TL;DR

This paper investigates how the self-attention mechanism in Transformer models limits their working memory capacity, drawing parallels with human cognition, and provides insights for designing models with improved memory capabilities.

Contribution

It demonstrates that attention score dispersion correlates with memory limits and offers a mechanistic explanation inspired by behavioral science theories.

Findings

Attention scores aggregate over training on N-back tasks.

Attention score entropy increases with N, indicating dispersion.

Self-attention may inherently constrain working memory in Transformers.

Abstract

Recent work on Transformer-based large language models (LLMs) has revealed striking limits in their working memory capacity, similar to what has been found in human behavioral studies. Specifically, these models' performance drops significantly on N-back tasks as N increases. However, there is still a lack of mechanistic interpretability as to why this phenomenon would arise. Inspired by the executive attention theory from behavioral sciences, we hypothesize that the self-attention mechanism within Transformer-based models might be responsible for their working memory capacity limits. To test this hypothesis, we train vanilla decoder-only transformers to perform N-back tasks and find that attention scores gradually aggregate to the N-back positions over training, suggesting that the model masters the task by learning a strategy to pay attention to the relationship between the current position and the N-back position. Critically, we find that the total entropy of the attention score matrix increases as N increases, suggesting that the dispersion of attention scores might be the cause of the capacity limit observed in N-back tasks. Our findings thus offer insights into the shared role of attention in both human and artificial intelligence. Moreover, the limitations of the self-attention mechanism revealed in the current study could inform future efforts to design more powerful model architectures with enhanced working memory capacity and cognitive capabilities.