Time to Spike? Understanding the Representational Power of Spiking Neural Networks in Discrete Time

TL;DR

This paper provides a theoretical analysis of discrete-time leaky integrate-and-fire spiking neural networks, revealing their function approximation capabilities, the influence of latency and depth, and contrasting them with traditional neural networks.

Contribution

It offers the first rigorous theoretical framework for understanding the representational power of discrete-time LIF-SNNs, including their function approximation and complexity.

Findings

LIF-SNNs realize piecewise constant functions on polyhedral regions.

Network size needed for function approximation is quantified.

Latency and depth significantly affect input space partitioning.

Abstract

Recent years have seen significant progress in developing spiking neural networks (SNNs) as a potential solution to the energy challenges posed by conventional artificial neural networks (ANNs). However, our theoretical understanding of SNNs remains relatively limited compared to the ever-growing body of literature on ANNs. In this paper, we study a discrete-time model of SNNs based on leaky integrate-and-fire (LIF) neurons, referred to as discrete-time LIF-SNNs, a widely used framework that still lacks solid theoretical foundations. We demonstrate that discrete-time LIF-SNNs with static inputs and outputs realize piecewise constant functions defined on polyhedral regions, and more importantly, we quantify the network size required to approximate continuous functions. Moreover, we investigate the impact of latency (number of time steps) and depth (number of layers) on the complexity of the input space partitioning induced by discrete-time LIF-SNNs. Our analysis highlights the importance of latency and contrasts these networks with ANNs employing piecewise linear activation functions. Finally, we present numerical experiments to support our theoretical findings.