The Role of Temporal Hierarchy in Spiking Neural Networks

TL;DR

This paper explores how imposing a hierarchy of temporal representations in Spiking Neural Networks enhances their ability to process temporal signals, leading to improved accuracy and naturally emerging optimized time constants.

Contribution

It introduces a hierarchical bias in the temporal parameters of SNNs, inspired by neuroscience, and demonstrates its benefits in both feed-forward and temporal convolutional SNNs.

Findings

Hierarchical temporal bias improves classification accuracy by up to 4.1%.

Hierarchical structure naturally emerges when optimizing time constants.

Enhanced performance on temporal spike-based datasets.

Abstract

Spiking Neural Networks (SNNs) have the potential for rich spatio-temporal signal processing thanks to exploiting both spatial and temporal parameters. The temporal dynamics such as time constants of the synapses and neurons and delays have been recently shown to have computational benefits that help reduce the overall number of parameters required in the network and increase the accuracy of the SNNs in solving temporal tasks. Optimizing such temporal parameters, for example, through gradient descent, gives rise to a temporal architecture for different problems. As has been shown in machine learning, to reduce the cost of optimization, architectural biases can be applied, in this case in the temporal domain. Such inductive biases in temporal parameters have been found in neuroscience studies, highlighting a hierarchy of temporal structure and input representation in different layers of the cortex. Motivated by this, we propose to impose a hierarchy of temporal representation in the hidden layers of SNNs, highlighting that such an inductive bias improves their performance. We demonstrate the positive effects of temporal hierarchy in the time constants of feed-forward SNNs applied to temporal tasks (Multi-Time-Scale XOR and Keyword Spotting, with a benefit of up to 4.1% in classification accuracy). Moreover, we show that such architectural biases, i.e. hierarchy of time constants, naturally emerge when optimizing the time constants through gradient descent, initialized as homogeneous values. We further pursue this proposal in temporal convolutional SNNs, by introducing the hierarchical bias in the size and dilation of temporal kernels, giving rise to competitive results in popular temporal spike-based datasets.