Krause Synchronization Transformers

TL;DR

Krause Attention introduces a localized, sparse attention mechanism inspired by consensus dynamics, improving efficiency and performance across vision and language models by mitigating global synchronization issues.

Contribution

The paper proposes Krause Attention, a novel attention mechanism that promotes local synchronization, reduces complexity, and enhances model performance across diverse tasks.

Findings

Krause Attention reduces runtime complexity from quadratic to linear.

It achieves consistent performance gains in vision and language models.

It alleviates attention sink phenomena and representation collapse.

Abstract

Self-attention in Transformers relies on globally normalized softmax weights, causing all tokens to compete for influence at every layer. When composed across depth, this interaction pattern induces strong synchronization dynamics that favor convergence toward a dominant mode, a behavior associated with representation collapse and attention sink phenomena. We introduce Krause Attention, a principled attention mechanism inspired by bounded-confidence consensus dynamics. Krause Attention replaces similarity-based global aggregation with distance-based, localized, and selectively sparse interactions, promoting structured local synchronization instead of global mixing. We relate this behavior to recent theory modeling Transformer dynamics as interacting particle systems, and show how bounded-confidence interactions naturally moderate attention concentration and alleviate attention sinks. Restricting interactions to local neighborhoods also reduces runtime complexity from quadratic to linear in sequence length. Empirically, we validate Krause Attention across diverse settings, including vision (ViT on CIFAR/ImageNet), autoregressive image generation (MNIST/CIFAR-10), large language models (Llama/Qwen), and language models trained from scratch at multiple scales (100M/200M). Across these domains, Krause Attention achieves consistent performance gains while improving computational efficiency, highlighting bounded-confidence dynamics as a scalable and effective inductive bias for attention.