Modern Neuromorphic AI: From Intra-Token to Inter-Token Processing

TL;DR

This paper explores how neuromorphic principles are integrated into modern AI architectures, focusing on intra-token and inter-token processing, and reviews training methods for energy-efficient, brain-inspired models.

Contribution

It systematically connects neuromorphic models with state-space and transformer architectures, highlighting recent advances in inter-token processing techniques.

Findings

Modern neuromorphic AI incorporates quantized activations and sparse attention.

Recent approaches leverage state-space dynamics for efficient inter-token processing.

Training methods include surrogate gradients and reinforcement learning.

Abstract

The rapid growth of artificial intelligence (AI) has brought novel data processing and generative capabilities but also escalating energy requirements. This challenge motivates renewed interest in neuromorphic computing principles, which promise brain-like efficiency through discrete and sparse activations, recurrent dynamics, and non-linear feedback. In fact, modern AI architectures increasingly embody neuromorphic principles through heavily quantized activations, state-space dynamics, and sparse attention mechanisms. This paper elaborates on the connections between neuromorphic models, state-space models, and transformer architectures through the lens of the distinction between intra-token processing and inter-token processing. Most early work on neuromorphic AI was based on spiking neural networks (SNNs) for intra-token processing, i.e., for transformations involving multiple channels, or features, of the same vector input, such as the pixels of an image. In contrast, more recent research has explored how neuromorphic principles can be leveraged to design efficient inter-token processing methods, which selectively combine different information elements depending on their contextual relevance. Implementing associative memorization mechanisms, these approaches leverage state-space dynamics or sparse self-attention. Along with a systematic presentation of modern neuromorphic AI models through the lens of intra-token and inter-token processing, training methodologies for neuromorphic AI models are also reviewed. These range from surrogate gradients leveraging parallel convolutional processing to local learning rules based on reinforcement learning mechanisms.