Toward Efficient Spiking Transformers: Synapse Pruning Meets Synergistic Learning-Based Compensation

TL;DR

This paper introduces a combined approach of synapse pruning and a novel neuron model to create lightweight, efficient spiking Transformer models that maintain high performance with reduced computational costs.

Contribution

It proposes tailored pruning strategies and a new neuron model to significantly reduce model size and complexity while preserving performance in spiking Transformers.

Findings

Model size and computational costs are substantially reduced.

Performance remains competitive despite pruning.

Pruning and compensation strategies are effective in efficient model design.

Abstract

As a foundational architecture of artificial intelligence models, Transformer has been recently adapted to spiking neural networks with promising performance across various tasks. However, existing spiking Transformer(ST)-based models require a substantial number of parameters and incur high computational costs, thus limiting their deployment in resource-constrained environments. To address these challenges, we propose combining synapse pruning with a synergistic learning-based compensation strategy to derive lightweight ST-based models. Specifically, two types of tailored pruning strategies are introduced to reduce redundancy in the weight matrices of ST blocks: an unstructured $\mathrm{L_{1}P}$ method to induce sparse representations, and a structured DSP method to induce low-rank representations. In addition, we propose an enhanced spiking neuron model, termed the synergistic leaky integrate-and-fire (sLIF) neuron, to effectively compensate for model pruning through synergistic learning between synaptic and intrinsic plasticity mechanisms. Extensive experiments on benchmark datasets demonstrate that the proposed methods significantly reduce model size and computational overhead while maintaining competitive performance. These results validate the effectiveness of the proposed pruning and compensation strategies in constructing efficient and high-performing ST-based models.