2D versus 3D Convolutional Spiking Neural Networks Trained with Unsupervised STDP for Human Action Recognition

TL;DR

This paper investigates 2D and 3D convolutional spiking neural networks trained with unsupervised STDP for human action recognition, demonstrating that 3D convolutional SNNs outperform 2D ones, especially on longer videos.

Contribution

Introduces a multi-layered 3D convolutional SNN trained with STDP for action recognition, comparing its performance to 2D models and showing the advantages of 3D convolutions.

Findings

3D convolutional SNNs learn motion patterns effectively.

3D convolution outperforms 2D in long video sequences.

STDP-based SNNs are promising for real-time video analysis.

Abstract

Current advances in technology have highlighted the importance of video analysis in the domain of computer vision. However, video analysis has considerably high computational costs with traditional artificial neural networks (ANNs). Spiking neural networks (SNNs) are third generation biologically plausible models that process the information in the form of spikes. Unsupervised learning with SNNs using the spike timing dependent plasticity (STDP) rule has the potential to overcome some bottlenecks of regular artificial neural networks, but STDP-based SNNs are still immature and their performance is far behind that of ANNs. In this work, we study the performance of SNNs when challenged with the task of human action recognition, because this task has many real-time applications in computer vision, such as video surveillance. In this paper we introduce a multi-layered 3D convolutional SNN model trained with unsupervised STDP. We compare the performance of this model to those of a 2D STDP-based SNN when challenged with the KTH and Weizmann datasets. We also compare single-layer and multi-layer versions of these models in order to get an accurate assessment of their performance. We show that STDP-based convolutional SNNs can learn motion patterns using 3D kernels, thus enabling motion-based recognition from videos. Finally, we give evidence that 3D convolution is superior to 2D convolution with STDP-based SNNs, especially when dealing with long video sequences.