EEvAct: Early Event-Based Action Recognition with High-Rate Two-Stream Spiking Neural Networks

TL;DR

This paper introduces a high-rate two-stream spiking neural network for early human activity recognition using event-based sensors, achieving improved accuracy and demonstrating real-world application in sports motion capture.

Contribution

The work presents a novel high-rate two-stream SNN framework that outperforms previous methods in early activity recognition accuracy on a large-scale dataset.

Findings

Outperforms previous methods by 2% in final accuracy

Provides a benchmark with Top-1 and Top-5 scores over observation time

Demonstrates effectiveness in real-world sports motion capture

Abstract

Recognizing human activities early is crucial for the safety and responsiveness of human-robot and human-machine interfaces. Due to their high temporal resolution and low latency, event-based vision sensors are a perfect match for this early recognition demand. However, most existing processing approaches accumulate events to low-rate frames or space-time voxels which limits the early prediction capabilities. In contrast, spiking neural networks (SNNs) can process the events at a high-rate for early predictions, but most works still fall short on final accuracy. In this work, we introduce a high-rate two-stream SNN which closes this gap by outperforming previous work by 2% in final accuracy on the large-scale THU EACT-50 dataset. We benchmark the SNNs within a novel early event-based recognition framework by reporting Top-1 and Top-5 recognition scores for growing observation time. Finally, we exemplify the impact of these methods on a real-world task of early action triggering for human motion capture in sports.