SNN-Driven Multimodal Human Action Recognition via Sparse Spatial-Temporal Data Fusion

TL;DR

This paper introduces a novel SNN-based framework for multimodal human action recognition using event camera and skeleton data, achieving high accuracy and energy efficiency suitable for resource-limited scenarios.

Contribution

The paper presents a new SNN architecture with modality-specific backbones and a discretized information bottleneck for efficient multimodal data fusion, advancing resource-efficient action recognition.

Findings

Achieves superior recognition accuracy compared to existing methods.

Demonstrates significant energy efficiency improvements.

Validates effectiveness on a newly constructed multimodal dataset.

Abstract

Multimodal human action recognition based on RGB and skeleton data fusion, while effective, is constrained by significant limitations such as high computational complexity, excessive memory consumption, and substantial energy demands, particularly when implemented with Artificial Neural Networks (ANN). These limitations restrict its applicability in resource-constrained scenarios. To address these challenges, we propose a novel Spiking Neural Network (SNN)-driven framework for multimodal human action recognition, utilizing event camera and skeleton data. Our framework is centered on two key innovations: (1) a novel multimodal SNN architecture that employs distinct backbone networks for each modality-an SNN-based Mamba for event camera data and a Spiking Graph Convolutional Network (SGN) for skeleton data-combined with a spiking semantic extraction module to capture deep semantic representations; and (2) a pioneering SNN-based discretized information bottleneck mechanism for modality fusion, which effectively balances the preservation of modality-specific semantics with efficient information compression. To validate our approach, we propose a novel method for constructing a multimodal dataset that integrates event camera and skeleton data, enabling comprehensive evaluation. Extensive experiments demonstrate that our method achieves superior performance in both recognition accuracy and energy efficiency, offering a promising solution for practical applications.