Expressivity of Spiking Neural Networks

TL;DR

This paper investigates the expressive capabilities of spiking neural networks, demonstrating their ability to realize both continuous and discontinuous functions and comparing their complexity bounds to traditional neural networks.

Contribution

It provides a theoretical analysis of the expressive power of spiking neural networks, including complexity bounds and their ability to emulate ReLU networks.

Findings

Spiking neural networks can realize both continuous and discontinuous functions.

Complexity bounds are established for emulating multi-layer ReLU networks.

Spiking neural networks can realize piecewise linear mappings similar to artificial neural networks.

Abstract

The synergy between spiking neural networks and neuromorphic hardware holds promise for the development of energy-efficient AI applications. Inspired by this potential, we revisit the foundational aspects to study the capabilities of spiking neural networks where information is encoded in the firing time of neurons. Under the Spike Response Model as a mathematical model of a spiking neuron with a linear response function, we compare the expressive power of artificial and spiking neural networks, where we initially show that they realize piecewise linear mappings. In contrast to ReLU networks, we prove that spiking neural networks can realize both continuous and discontinuous functions. Moreover, we provide complexity bounds on the size of spiking neural networks to emulate multi-layer (ReLU) neural networks. Restricting to the continuous setting, we also establish complexity bounds in the reverse direction for one-layer spiking neural networks.