Temporal-wise Attention Spiking Neural Networks for Event Streams Classification

TL;DR

This paper introduces TA-SNN, a novel spiking neural network model with temporal-wise attention that enhances event stream classification by focusing on significant frames, achieving state-of-the-art results across multiple tasks.

Contribution

The paper proposes a temporal-wise attention mechanism for SNNs to better handle sparse, non-uniform event streams, improving classification accuracy and efficiency.

Findings

Achieved nearly 19% accuracy improvement in gesture recognition.

Demonstrated state-of-the-art results on three event stream classification tasks.

Validated effectiveness across gesture, image, and spoken digit recognition.

Abstract

How to effectively and efficiently deal with spatio-temporal event streams, where the events are generally sparse and non-uniform and have the microsecond temporal resolution, is of great value and has various real-life applications. Spiking neural network (SNN), as one of the brain-inspired event-triggered computing models, has the potential to extract effective spatio-temporal features from the event streams. However, when aggregating individual events into frames with a new higher temporal resolution, existing SNN models do not attach importance to that the serial frames have different signal-to-noise ratios since event streams are sparse and non-uniform. This situation interferes with the performance of existing SNNs. In this work, we propose a temporal-wise attention SNN (TA-SNN) model to learn frame-based representation for processing event streams. Concretely, we extend the attention concept to temporal-wise input to judge the significance of frames for the final decision at the training stage, and discard the irrelevant frames at the inference stage. We demonstrate that TA-SNN models improve the accuracy of event streams classification tasks. We also study the impact of multiple-scale temporal resolutions for frame-based representation. Our approach is tested on three different classification tasks: gesture recognition, image classification, and spoken digit recognition. We report the state-of-the-art results on these tasks, and get the essential improvement of accuracy (almost 19\%) for gesture recognition with only 60 ms.