Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse

TL;DR

This paper reveals that the attention sink phenomenon in large language models naturally forms a Mixture-of-Experts structure within attention layers, and proposes sink-aware training to mitigate head collapse and improve performance.

Contribution

It provides a theoretical and empirical analysis linking attention sink to MoE structures and introduces a sink-aware training method to address head collapse in attention layers.

Findings

Attention sink naturally constructs a Mixture-of-Experts within attention layers.

Sink-aware training improves head load balancing and model performance.

The method is effective across various attention mechanisms.

Abstract

Large Language Models (LLMs) often assign disproportionate attention to the first token, a phenomenon known as the attention sink. Several recent approaches aim to address this issue, including Sink Attention in GPT-OSS and Gated Attention in Qwen3-Next. However, a comprehensive analysis of the relationship among these attention mechanisms is lacking. In this work, we provide both theoretical and empirical evidence demonstrating that the sink in Vanilla Attention and Sink Attention naturally construct a Mixture-of-Experts (MoE) mechanism within attention layers. This insight explains the head collapse phenomenon observed in prior work, where only a fixed subset of attention heads contributes to generation. To mitigate head collapse, we propose a sink-aware training algorithm with an auxiliary load balancing loss designed for attention layers. Extensive experiments show that our method achieves effective head load balancing and improves model performance across Vanilla Attention, Sink Attention, and Gated Attention. We hope this study offers a new perspective on attention mechanisms and encourages further exploration of the inherent MoE structure within attention layers.