Hybrid Spiking Neural Network -- Transformer Video Classification Model

TL;DR

This paper proposes a novel hybrid spiking neural network architecture inspired by brain structures for video classification, combining SNNs with transformer models, and provides training procedures and open-source code.

Contribution

It introduces the first cortical column-like hybrid SNN-transformer model for time-series data classification, with new encoding methods and training procedures.

Findings

First hybrid SNN-transformer architecture for video classification

Developed new encoding methods for SNNs

Provided open-source implementation for easier adoption

Abstract

In recent years, Spiking Neural Networks (SNNs) have gathered significant interest due to their temporal understanding capabilities. This work introduces, to the best of our knowledge, the first Cortical Column like hybrid architecture for the Time-Series Data Classification Task that leverages SNNs and is inspired by the brain structure, inspired from the previous hybrid models. We introduce several encoding methods to use with this model. Finally, we develop a procedure for training this network on the training dataset. As an effort to make using these models simpler, we make all the implementations available to the public.