Kuramoto Oscillatory Phase Encoding: Neuro-inspired Synchronization for Improved Learning Efficiency

TL;DR

This paper introduces KoPE, a neuro-inspired phase synchronization method for Vision Transformers, enhancing learning efficiency and performance on structured vision tasks through oscillatory synchronization.

Contribution

It proposes a novel phase encoding mechanism inspired by neural oscillations to improve training efficiency and task performance in vision models.

Findings

KoPE improves training, parameter, and data efficiency in vision models.

KoPE enhances performance on structured vision tasks like segmentation and reasoning.

Theoretical and empirical results show KoPE accelerates attention concentration.

Abstract

Spatiotemporal neural dynamics and oscillatory synchronization are widely implicated in biological information processing and have been hypothesized to support flexible coordination such as feature binding. By contrast, most deep learning architectures represent and propagate information through activation values, neglecting the joint dynamics of rate and phase. In this work, we introduce Kuramoto oscillatory Phase Encoding (KoPE) as an additional, evolving phase state to Vision Transformers, incorporating a neuro-inspired synchronization mechanism to advance learning efficiency. We show that KoPE can improve training, parameter, and data efficiency of vision models through synchronization-enhanced structure learning. Moreover, KoPE benefits tasks requiring structured understanding, including semantic and panoptic segmentation, representation alignment with language, and few-shot abstract visual reasoning (ARC-AGI). Theoretical analysis and empirical verification further suggest that KoPE can accelerate attention concentration for learning efficiency. These results indicate that synchronization can serve as a scalable, neuro-inspired mechanism for advancing state-of-the-art neural network models.