ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition

TL;DR

ReSpike is a hybrid neural network framework combining ANNs and SNNs to improve action recognition in videos, achieving high accuracy and energy efficiency by decomposing video data into spatial and temporal components and using multi-scale cross-attention for feature fusion.

Contribution

The paper introduces ReSpike, a novel hybrid architecture that effectively combines ANNs and SNNs for video action recognition, with a multi-scale cross-attention mechanism for feature fusion.

Findings

>30% accuracy improvement over SNN baselines on benchmark datasets

Achieves comparable accuracy to traditional ANNs with better energy efficiency

Effective decomposition of video into spatial and temporal components enhances recognition performance

Abstract

Spiking Neural Networks (SNNs) have emerged as a compelling, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) for static image tasks such as image classification and segmentation. However, in the more complex video classification domain, SNN-based methods fall considerably short of ANN-based benchmarks due to the challenges in processing dense frame sequences. To bridge this gap, we propose ReSpike, a hybrid framework that synergizes the strengths of ANNs and SNNs to tackle action recognition tasks with high accuracy and low energy cost. By decomposing film clips into spatial and temporal components, i.e., RGB image Key Frames and event-like Residual Frames, ReSpike leverages ANN for learning spatial information and SNN for learning temporal information. In addition, we propose a multi-scale cross-attention mechanism for effective feature fusion. Compared to state-of-the-art SNN baselines, our ReSpike hybrid architecture demonstrates significant performance improvements (e.g., >30% absolute accuracy improvement on HMDB-51, UCF-101, and Kinetics-400). Furthermore, ReSpike achieves comparable performance with prior ANN approaches while bringing better accuracy-energy tradeoff.