Fractional neural attention for efficient multiscale sequence processing

TL;DR

This paper introduces Fractional Neural Attention (FNA), a neuroscience-inspired multiscale attention mechanism based on fractional diffusion, enhancing Transformer efficiency and interpretability across NLP, image processing, and translation tasks.

Contribution

FNA models token interactions via Lévy diffusion governed by the fractional Laplacian, offering a novel, biologically grounded approach to multiscale information processing in neural networks.

Findings

FNA achieves competitive text classification with a single layer and head.

FNA improves performance in image processing tasks.

FNA enables dimensionality reduction of weights while preserving structure.

Abstract

Attention mechanisms underpin the computational power of Transformer models, which have achieved remarkable success across diverse domains. Yet understanding and extending the principles underlying self-attention remains a key challenge for advancing artificial intelligence. Drawing inspiration from the multiscale dynamics of biological attention and from dynamical systems theory, we introduce Fractional Neural Attention (FNA), a principled, neuroscience-inspired framework for multiscale information processing. FNA models token interactions through L\'evy diffusion governed by the fractional Laplacian, intrinsically realizing simultaneous short- and long-range dependencies across multiple scales. This mechanism yields greater expressivity and faster information mixing, advancing the foundational capacity of Transformers. Theoretically, we show that FNA's dynamics are governed by the fractional diffusion equation, and that the resulting attention networks exhibit larger spectral gaps and shorter path lengths -- mechanistic signatures of enhanced computational efficiency. Empirically, FNA achieves competitive text-classification performance even with a single layer and a single head; it also improves performance in image processing and neural machine translation. Finally, the diffusion map algorithm from geometric harmonics enables dimensionality reduction of FNA weights while preserving the intrinsic structure of embeddings and hidden states. Together, these results establish FNA as a principled mechanism connecting self-attention, stochastic dynamics, and geometry, providing an interpretable, biologically grounded foundation for powerful, neuroscience-inspired AI.