
TL;DR
This paper introduces Selective Synchronization Attention (SSA), a biologically inspired, efficient attention mechanism based on coupled oscillators that naturally induces sparsity and unifies positional and semantic encoding.
Contribution
The paper proposes SSA, a novel attention mechanism derived from oscillator synchronization, replacing standard self-attention with a closed-form, biologically motivated approach that improves efficiency and interpretability.
Findings
SSA provides natural sparsity through phase-locking thresholds.
SSA unifies positional and semantic encoding via frequency spectrum.
Synchronization matrices reveal diverse, head-specific coupling patterns.
Abstract
The Transformer architecture has become the foundation of modern deep learning, yet its core self-attention mechanism suffers from quadratic computational complexity and lacks grounding in biological neural computation. We propose Selective Synchronization Attention (SSA), a novel attention mechanism that replaces the standard dot-product self-attention with a closed-form operator derived from the steady-state solution of the Kuramoto model of coupled oscillators. In SSA, each token is represented as an oscillator characterized by a learnable natural frequency and phase; the synchronization strength between token pairs, determined by a frequency-dependent coupling and phase-locking condition, serves as the attention weight. This formulation provides three key advantages: (i) natural sparsity arising from the phase-locking threshold, whereby tokens with incompatible frequencies…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNonlinear Dynamics and Pattern Formation · Neural Networks and Reservoir Computing · Neural dynamics and brain function
