Spectral Selection in Symmetric Self-Attention Dynamics

TL;DR

This paper analyzes the spectral dynamics of symmetric self-attention in Transformers, revealing how the spectrum of weight matrices influences asymptotic alignment or polarization of the system.

Contribution

It provides a rigorous finite-particle analysis of spectral mode selection in symmetric self-attention flows, connecting eigenstructure to asymptotic behavior.

Findings

Homogeneous alignment occurs when a dominant positive eigenvalue exists.

Sign-split polarization happens when the matrix is negative definite.

Local stability and global selection criteria are established for pure modes.

Abstract

We study self-attention dynamics on the unit sphere as an interacting particle system arising from an idealized Transformer-type update. Under a symmetry assumption on weight matrices given by $Q^\top K=V=V^\top$, the flow admits a gradient-flow structure and an exact reformulation in the eigenbasis of $V$, revealing a spectral mode-selection mechanism. We show that the dynamics exhibits two distinct asymptotic scenarios: homogeneous alignment toward the dominant eigendirection when one positive eigenvalue strictly dominates all others in modulus, and sign-split polarization toward the most negative eigendirection when $V$ is negative definite. In particular, we obtain local stability criteria for pure-mode equilibria and global selection results in both regimes. These results provide a rigorous finite-particle description of how the spectrum of the weight matrices organizes asymptotic patterns in a symmetric self-attention flow, and highlight how the symmetric setting renders the dynamics amenable to mathematical analysis.