Artificial Kuramoto Oscillatory Neurons
Takeru Miyato, Sindy L\"owe, Andreas Geiger, Max Welling
TL;DR
This paper introduces Artificial Kuramoto Oscillatory Neurons (AKOrN), a dynamical neuron model based on synchronization, which improves performance across various AI tasks by emphasizing the importance of dynamic representations.
Contribution
The paper presents AKOrN as a novel dynamical neuron model that can be integrated with various network architectures, demonstrating improved task performance.
Findings
Enhanced unsupervised object discovery
Improved adversarial robustness
Better calibrated uncertainty quantification
Abstract
It has long been known in both neuroscience and AI that ``binding'' between neurons leads to a form of competitive learning where representations are compressed in order to represent more abstract concepts in deeper layers of the network. More recently, it was also hypothesized that dynamic (spatiotemporal) representations play an important role in both neuroscience and AI. Building on these ideas, we introduce Artificial Kuramoto Oscillatory Neurons (AKOrN) as a dynamical alternative to threshold units, which can be combined with arbitrary connectivity designs such as fully connected, convolutional, or attentive mechanisms. Our generalized Kuramoto updates bind neurons together through their synchronization dynamics. We show that this idea provides performance improvements across a wide spectrum of tasks such as unsupervised object discovery, adversarial robustness, calibrated…
Peer Reviews
Decision·ICLR 2025 Oral
1. The neuronal architecture draws inspiration from physics and neuroscience and is very novel in deep learning. 2. Model performance is impressive across a range of tasks. 3. Extensive experiments show that the method applies to mainstream architectures including convolutional neural networks and transformers.
1. While AKOrN shows promising performance on the selected tasks, the model might not work as well on more "classical tasks" such as image classification. 2. Although the motivation of AKOrN is clear from a neuroscience perspective, how, and why the model shows superior performance in the tested task is not well understood.
- the model is interesting and well defined - the quantitative results appear impressive, though I am have some issues with their presentation (see below) - the many numerical results presented, and comparisons to other models, suggests that overall a good deal of thought/time has been put in the manuscript
- there are numerous minor issues with the text/presentation (see question below) which impacts the clarity of the paper - This not a field I have experience in, but I am not entirley convinced a fair comparison was made to other models for object discovery. Is the idea that AKOrN is a highly competitive model for models 'trained from scratch', and that comparing it to models with some pretrained parameters (e.g. Lowe et al. 2024) is unfair? I think this could be better clarified if so - there
I enjoyed reading this paper, impressed by the achievements on diverse tasks and also the authors's thoughts dispersed over the paper. In brief, the strength of the paper includes: - Soft grouping/clustering in neural network is an important question and often missing in main-stream ANNs. - The proposed neuron model is neuroscientific motivated yet has concise fomula. The physical origin of the model makes it plausible to construct an "energy function", which turns out to be quite informative o
For the weakness: - It is not clear what is the actual conceptual contribution / nolvelty of the Kura model proposed in this paper compared with Lowe's Rotating Feature (and recent updates). For example, equation (6) is very similiar to the "binding mechanism activation" in [1] and I have the intuition that this is the essential mechanism for the binding ability of model, similiar to [1]. If I am wrong, please correct me. - When generalizing the original Kura model to high-dim, it is not clear
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNonlinear Dynamics and Pattern Formation