SPACT18: Spiking Human Action Recognition Benchmark Dataset with Complementary RGB and Thermal Modalities

TL;DR

This paper introduces SPACT18, a novel multimodal dataset with spike camera, RGB, and thermal data for benchmarking spiking neural networks in action recognition, promoting energy-efficient video understanding.

Contribution

It provides the first spike camera-based action recognition dataset with synchronized RGB and thermal modalities for comprehensive benchmarking.

Findings

Dataset enables multimodal video understanding research.

Preserves sparsity and temporal precision of spiking data.

Facilitates comparison of spiking, thermal, and RGB modalities.

Abstract

Spike cameras, bio-inspired vision sensors, asynchronously fire spikes by accumulating light intensities at each pixel, offering ultra-high energy efficiency and exceptional temporal resolution. Unlike event cameras, which record changes in light intensity to capture motion, spike cameras provide even finer spatiotemporal resolution and a more precise representation of continuous changes. In this paper, we introduce the first video action recognition (VAR) dataset using spike camera, alongside synchronized RGB and thermal modalities, to enable comprehensive benchmarking for Spiking Neural Networks (SNNs). By preserving the inherent sparsity and temporal precision of spiking data, our three datasets offer a unique platform for exploring multimodal video understanding and serve as a valuable resource for directly comparing spiking, thermal, and RGB modalities. This work contributes a novel dataset that will drive research in energy-efficient, ultra-low-power video understanding, specifically for action recognition tasks using spike-based data.