How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective

TL;DR

This paper investigates how attention sinks, especially at the first token, form in large language models, revealing a simple circuit mechanism and early training emergence that impacts interpretability and training analysis.

Contribution

It introduces the P0 Sink Circuit as a mechanism for attention sinks at position zero and shows its early emergence during training in large models.

Findings

Attention sinks at position zero are driven by a simple circuit mechanism.

The P0 Sink Circuit emerges early in training and concentrates in initial layers.

This mechanism can serve as a signal for training convergence.

Abstract

Large Language Models (LLMs) often allocate disproportionate attention to specific tokens, a phenomenon commonly referred to as the attention sink. While such sinks are generally considered detrimental, prior studies have identified a notable exception: the model's consistent emphasis on the first token of the input sequence. This structural bias can influence a wide range of downstream applications and warrants careful consideration. Despite its prevalence, the precise mechanisms underlying the emergence and persistence of attention sinks remain poorly understood. In this work, we trace the formation of attention sinks around the first token of the input. We identify a simple mechanism, referred to as the P0 Sink Circuit, that enables the model to recognize token at position zero and induce an attention sink within two transformer blocks, without relying on any semantic information. This mechanism serves as the basis for the attention sink on position zero. Furthermore, by analyzing training traces from a 30B A3B MoE model trained from scratch, we find that this mechanism emerges early in training and becomes increasingly concentrated in the first two layers, suggesting a possible signal for tracking pre training convergence states.