Cannistraci-Hebb Training on Ultra-Sparse Spiking Neural Networks

TL;DR

This paper introduces CH-SNN, a dynamic sparse training framework for spiking neural networks that maintains high performance at ultra-sparse levels, inspired by brain-like learning mechanisms.

Contribution

It proposes a novel four-stage sparse training method incorporating topological initialization, weight sparsity, pruning, and link prediction based on Cannistraci-Hebb theory.

Findings

Achieves ultra-sparse SNNs with minimal performance loss

Demonstrates effectiveness across multiple datasets and architectures

Enables energy-efficient neuromorphic computing applications

Abstract

Inspired by the brain's spike-based computation, spiking neural networks (SNNs) inherently possess temporal activation sparsity. However, when it comes to the sparse training of SNNs in the structural connection domain, existing methods fail to achieve ultra-sparse network structures without significant performance loss, thereby hindering progress in energy-efficient neuromorphic computing. This limitation presents a critical challenge: how to achieve high levels of structural connection sparsity while maintaining performance comparable to fully connected networks. To address this challenge, we propose the Cannistraci-Hebb Spiking Neural Network (CH-SNN), a novel and generalizable dynamic sparse training framework for SNNs consisting of four stages. First, we propose a sparse spike correlated topological initialization (SSCTI) method to initialize a sparse network based on node correlations. Second, temporal activation sparsity and structural connection sparsity are integrated via a proposed sparse spike weight initialization (SSWI) method. Third, a hybrid link removal score (LRS) is applied to prune redundant weights and inactive neurons, improving information flow. Finally, the CH3-L3 network automaton framework inspired by Cannistraci-Hebb learning theory is incorporated to perform link prediction for potential synaptic regrowth. These mechanisms enable CH-SNN to achieve sparsification across all linear layers. We have conducted extensive experiments on six datasets including CIFAR-10 and CIFAR-100, evaluating various network architectures such as spiking convolutional neural networks and Spikformer.