Selective Synchronization Attention

TL;DR

This paper introduces Selective Synchronization Attention (SSA), a biologically inspired, efficient attention mechanism based on coupled oscillators that naturally induces sparsity and unifies positional and semantic encoding.

Contribution

The paper proposes SSA, a novel attention mechanism derived from oscillator synchronization, replacing standard self-attention with a closed-form, biologically motivated approach that improves efficiency and interpretability.

Findings

SSA provides natural sparsity through phase-locking thresholds.

SSA unifies positional and semantic encoding via frequency spectrum.

Synchronization matrices reveal diverse, head-specific coupling patterns.

Abstract

The Transformer architecture has become the foundation of modern deep learning, yet its core self-attention mechanism suffers from quadratic computational complexity and lacks grounding in biological neural computation. We propose Selective Synchronization Attention (SSA), a novel attention mechanism that replaces the standard dot-product self-attention with a closed-form operator derived from the steady-state solution of the Kuramoto model of coupled oscillators. In SSA, each token is represented as an oscillator characterized by a learnable natural frequency and phase; the synchronization strength between token pairs, determined by a frequency-dependent coupling and phase-locking condition, serves as the attention weight. This formulation provides three key advantages: (i) natural sparsity arising from the phase-locking threshold, whereby tokens with incompatible frequencies automatically receive zero attention weight without explicit masking; (ii) unified positional-semantic encoding through the natural frequency spectrum, eliminating the need for separate positional encodings; and (iii) a single-pass, closed-form computation that avoids iterative ODE integration, with all components (coupling, order parameter, synchronization) derived from the oscillatory framework. We instantiate SSA within the Oscillatory Synchronization Network (OSN), a drop-in replacement for the Transformer block. Analysis of the synchronization matrices reveals non-uniform, head-diverse coupling patterns even at initialization, demonstrating a stronger architectural inductive bias than the approximately uniform attention produced by randomly initialized Transformers.