Probabilistic Modeling: Proving the Lottery Ticket Hypothesis in Spiking Neural Network

TL;DR

This paper extends the Lottery Ticket Hypothesis to Spiking Neural Networks by developing a probabilistic model that accounts for their discontinuous dynamics, providing theoretical proof and improved pruning methods.

Contribution

It introduces a novel probabilistic modeling approach for SNNs and proves LTH applicability, overcoming previous limitations due to non-Lipschitz spiking functions.

Findings

LTH holds in SNNs under the proposed probabilistic model

Pruning based on weight size is suboptimal for SNNs

New pruning criterion improves performance over baseline

Abstract

The Lottery Ticket Hypothesis (LTH) states that a randomly-initialized large neural network contains a small sub-network (i.e., winning tickets) which, when trained in isolation, can achieve comparable performance to the large network. LTH opens up a new path for network pruning. Existing proofs of LTH in Artificial Neural Networks (ANNs) are based on continuous activation functions, such as ReLU, which satisfying the Lipschitz condition. However, these theoretical methods are not applicable in Spiking Neural Networks (SNNs) due to the discontinuous of spiking function. We argue that it is possible to extend the scope of LTH by eliminating Lipschitz condition. Specifically, we propose a novel probabilistic modeling approach for spiking neurons with complicated spatio-temporal dynamics. Then we theoretically and experimentally prove that LTH holds in SNNs. According to our theorem, we conclude that pruning directly in accordance with the weight size in existing SNNs is clearly not optimal. We further design a new criterion for pruning based on our theory, which achieves better pruning results than baseline.