TP-Spikformer: Token Pruned Spiking Transformer

TL;DR

This paper introduces TP-Spikformer, a token pruning method for spiking transformers that reduces computational costs while maintaining performance, enabling more efficient deployment of energy-efficient spiking neural networks.

Contribution

The paper proposes a novel token pruning framework with a heuristic importance criterion and early stopping strategy, improving efficiency without sacrificing accuracy in spiking transformers.

Findings

Effective reduction in computational overhead across multiple architectures.

Maintains competitive accuracy in diverse tasks.

Performs well in a training-free manner.

Abstract

Spiking neural networks (SNNs) offer an energy-efficient alternative to traditional neural networks due to their event-driven computing paradigm. However, recent advancements in spiking transformers have focused on improving accuracy with large-scale architectures, which require significant computational resources and limit deployment on resource-constrained devices. In this paper, we propose a simple yet effective token pruning method for spiking transformers, termed TP-Spikformer, that reduces storage and computational overhead while maintaining competitive performance. Specifically, we first introduce a heuristic spatiotemporal information-retaining criterion that comprehensively evaluates tokens' importance, assigning higher scores to informative tokens for retention and lower scores to uninformative ones for pruning. Based on this criterion, we propose an information-retaining token pruning framework that employs a block-level early stopping strategy for uninformative tokens, instead of removing them outright. This also helps preserve more information during token pruning. We demonstrate the effectiveness, efficiency and scalability of TP-Spikformer through extensive experiments across diverse architectures, including Spikformer, QKFormer and Spike-driven Transformer V1 and V3, and a range of tasks such as image classification, object detection, semantic segmentation and event-based object tracking. Particularly, TP-Spikformer performs well in a training-free manner. These results reveal its potential as an efficient and practical solution for deploying SNNs in real-world applications with limited computational resources.