Elastic Spiking Transformers for Efficient Gesture Understanding

TL;DR

This paper introduces Elastic Spiking Transformers, a flexible, runtime-adaptive neural architecture for energy-efficient gesture recognition on neuromorphic hardware, capable of adjusting complexity dynamically without retraining.

Contribution

The paper presents a novel elastic architecture for spiking transformers that enables dynamic adjustment of network size and attention heads at inference time, improving deployment flexibility and energy efficiency.

Findings

Achieves comparable or better accuracy than static models on multiple datasets.

Supports real-time gesture recognition on resource-constrained devices.

Reduces spike firing rates proportionally with network size adjustments.

Abstract

Spiking Neural Networks (SNNs), particularly Spiking Transformers, offer energy-efficient processing of event-based sensor data for healthcare applications. Yet current architectures are rigid: they are trained and deployed as static networks with fixed parameter counts and computational graphs. This limits deployment on neuromorphic hardware such as Loihi and SpiNNaker, where on-chip constraints often require smaller models that trade accuracy for feasibility. We introduce the Elastic Spiking Transformer, a runtime-adaptive architecture that brings elasticity into the spiking paradigm. Inspired by Matryoshka-style representation learning, it embeds nested elasticity in the Feature Extractor, Spiking Self-Attention, and Feed-Forward blocks. Through granularity-aware weight sharing, a single universal model can dynamically slice network width and attention heads at inference time without retraining. This design provides two key advantages for SNNs. First, it allows the model to adjust its parameter footprint to different hardware memory budgets. Second, reducing active neurons also lowers spike firing rates, yielding proportional reductions in synaptic operations, an energy benefit not directly available in standard artificial neural networks. We evaluate the approach on CIFAR10/100, CIFAR10-DVS, and the EHWGesture clinical gesture understanding dataset. Results show that one Elastic Spiking Transformer spans a broad range of complexity-accuracy trade-offs, matching or surpassing independently trained baselines while supporting adaptive, real-time gesture recognition on resource-constrained edge devices.