Empirical study on the efficiency of Spiking Neural Networks with axonal delays, and algorithm-hardware benchmarking

TL;DR

This paper investigates how axonal delays in Spiking Neural Networks affect performance and efficiency, demonstrating state-of-the-art accuracy with reduced parameters and energy consumption on benchmark datasets.

Contribution

It introduces a methodology to incorporate axonal delays into training of SNNs and empirically evaluates their impact on performance and hardware efficiency.

Findings

SNNs with axonal delays achieve over 90% accuracy on SHD dataset.

Delay-based models require less than half the trainable synapses of traditional models.

Parameter reduction leads to approximately 90% savings in energy and memory.

Abstract

The role of axonal synaptic delays in the efficacy and performance of artificial neural networks has been largely unexplored. In step-based analog-valued neural network models (ANNs), the concept is almost absent. In their spiking neuroscience-inspired counterparts, there is hardly a systematic account of their effects on model performance in terms of accuracy and number of synaptic operations.This paper proposes a methodology for accounting for axonal delays in the training loop of deep Spiking Neural Networks (SNNs), intending to efficiently solve machine learning tasks on data with rich temporal dependencies. We then conduct an empirical study of the effects of axonal delays on model performance during inference for the Adding task, a benchmark for sequential regression, and for the Spiking Heidelberg Digits dataset (SHD), commonly used for evaluating event-driven models. Quantitative results on the SHD show that SNNs incorporating axonal delays instead of explicit recurrent synapses achieve state-of-the-art, over 90% test accuracy while needing less than half trainable synapses. Additionally, we estimate the required memory in terms of total parameters and energy consumption of accomodating such delay-trained models on a modern neuromorphic accelerator. These estimations are based on the number of synaptic operations and the reference GF-22nm FDX CMOS technology. As a result, we demonstrate that a reduced parameterization, which incorporates axonal delays, leads to approximately 90% energy and memory reduction in digital hardware implementations for a similar performance in the aforementioned task.