Digit Recognition using Multimodal Spiking Neural Networks

TL;DR

This paper demonstrates that multimodal spiking neural networks, combining visual and auditory event-based data, outperform unimodal networks in digit classification tasks, achieving over 98% accuracy.

Contribution

The work introduces a multimodal SNN architecture that fuses visual and auditory inputs at various depths, showing improved performance over unimodal approaches.

Findings

Multimodal SNNs outperform unimodal SNNs in digit classification.

Fusion at any depth yields similar performance improvements.

Achieved 98.43% accuracy on combined event-based datasets.

Abstract

Spiking neural networks (SNNs) are the third generation of neural networks that are biologically inspired to process data in a fashion that emulates the exchange of signals in the brain. Within the Computer Vision community SNNs have garnered significant attention due in large part to the availability of event-based sensors that produce a spatially resolved spike train in response to changes in scene radiance. SNNs are used to process event-based data due to their neuromorphic nature. The proposed work examines the neuromorphic advantage of fusing multiple sensory inputs in classification tasks. Specifically we study the performance of a SNN in digit classification by passing in a visual modality branch (Neuromorphic-MNIST [N-MNIST]) and an auditory modality branch (Spiking Heidelberg Digits [SHD]) from datasets that were created using event-based sensors to generate a series of time-dependent events. It is observed that multi-modal SNNs outperform unimodal visual and unimodal auditory SNNs. Furthermore, it is observed that the process of sensory fusion is insensitive to the depth at which the visual and auditory branches are combined. This work achieves a 98.43% accuracy on the combined N-MNIST and SHD dataset using a multimodal SNN that concatenates the visual and auditory branches at a late depth.