Generalization Bounds of Spiking Neural Networks via Rademacher Complexity

TL;DR

This paper provides a theoretical analysis of the generalization bounds of Spiking Neural Networks (SNNs) using Rademacher complexity, revealing how various factors influence their ability to perform well on unseen data.

Contribution

It offers a more precise theoretical bound on SNN generalization, considering multiple configurations and integration schemes, improving understanding over prior bounds.

Findings

Empirical Rademacher complexity of SNNs is close to configurations, exponential to depth and duration.

Complexity is superlinear/subquadratic to network width and polynomial to parameter norm.

Bound is inverse-linear to training sample size and independent of neuron computations.

Abstract

Spiking Neural Networks (SNNs) have garnered increasing attention as one of bio-inspired models due to their great potential in neuromorphic computing and sparse computation. Many practical algorithms and techniques have been developed; however, theoretical understandings of the generalization, that is, the extent to which SNNs perform well on unseen data, are far from clear. Recent advances disclosed an excitation-dependent and architecture-related generalization bound such that the Rademacher complexity of SNNs with stochastic firing can be upper bounded by an exponential function relative to the excitation probability and the architecture depth. In this paper, we theoretically investigate the generalization bounds of SNNs with several integration-and-fire schemes via Rademacher complexity. We recognize that the empirical Rademacher complexity of SNNs is close to the SNN configurations, which is exponential to the network depth and the maximum time duration of received spike sequences, superlinear and subquadratic to the network width, polynomial to the parameter norm, inverse-linear to the number of training samples, and independent of the computations within spiking neurons, achieving a more precise rate than conventional studies. Our theoretical results may support the scope of SNN theories and shed some insight into the development of SNNs.