Uncertainty-Aware Token Importance Estimation in Spiking Transformers

TL;DR

This paper introduces Uncert, a novel, training-free method for estimating token importance in spiking transformers by analyzing temporal uncertainty patterns, leading to improved efficiency without sacrificing accuracy.

Contribution

Uncert leverages temporal uncertainty trajectories modeled by a Dirichlet distribution to effectively identify redundant tokens in spiking transformers, enhancing efficiency.

Findings

Uncert achieves better accuracy-efficiency tradeoffs in token pruning.

Temporal uncertainty patterns correlate with token contribution.

Uncert is training-free and plug-and-play for spiking transformers.

Abstract

Spiking transformers have shown strong potential for neuromorphic vision, yet their token processing across multiple spiking steps still introduces substantial redundancy and inference cost. Existing token reduction methods mainly rely on response based cues, such as activation magnitude, firing statistics, or feature similarity. Although effective, these criteria do not explicitly characterize token importance from the perspective of temporally evolving class evidence. In spiking transformers, token representations are progressively formed across multiple spiking steps rather than determined at a single instant, suggesting that token importance should be evaluated not only by instantaneous responses but also by temporal uncertainty patterns. Our key observation is that tokens exhibit heterogeneous uncertainty trajectories over time, and that their temporally aggregated uncertainty statistics provide an effective cue for distinguishing informative tokens from redundant ones. Motivated by this, we propose Uncert, a training free and plug and play token importance estimation framework for spiking transformers. Specifically, Uncert models token wise class evidence with a Dirichlet distribution and summarizes each token temporal uncertainty using its mean and fluctuation across spiking steps, yielding an uncertainty aware importance score for token reduction during inference. Experiments on both static and neuromorphic benchmarks show that Uncert achieves favorable accuracy and efficiency tradeoffs, with the most consistent gains observed under token pruning. Further analysis reveals a clear empirical connection between temporal uncertainty patterns and token contribution, offering new insights into token dynamics in spiking transformers.