Toward Relative Positional Encoding in Spiking Transformers

TL;DR

This paper introduces novel relative positional encoding strategies for spiking Transformers, improving their ability to model sequences and images while maintaining binary spike properties, and demonstrates their effectiveness across multiple tasks.

Contribution

The paper proposes Gray-PE and Log-PE methods for relative positional encoding in spiking Transformers, extending them to 2D for image processing, and validates their performance improvements.

Findings

Gray-PE maintains constant Hamming distance for positional indices differing by powers of two.

Log-PE integrates logarithmic relative distances into attention maps.

RPE methods enhance performance in time series, text, and image classification tasks.

Abstract

Spiking neural networks (SNNs) are bio-inspired networks that mimic how neurons in the brain communicate through discrete spikes, which have great potential in various tasks due to their energy efficiency and temporal processing capabilities. SNNs with self-attention mechanisms (spiking Transformers) have recently shown great advancements in various tasks, and inspired by traditional Transformers, several studies have demonstrated that spiking absolute positional encoding can help capture sequential relationships for input data, enhancing the capabilities of spiking Transformers for tasks such as sequential modeling and image classification. However, how to incorporate relative positional information into SNNs remains a challenge. In this paper, we introduce several strategies to approximate relative positional encoding in spiking Transformers while preserving the binary nature of spikes. Firstly, we formally prove that encoding relative distances with Gray Code ensures that the binary representations of positional indices maintain a constant Hamming distance whenever their decimal values differ by a power of two, and we propose Gray-PE based on this property. In addition, we propose another RPE method called Log-PE, which combines the logarithmic form of the relative distance matrix directly into the spiking attention map. Furthermore, we extend our RPE methods to a two-dimensional form, making them suitable for processing image patches. We evaluate our RPE methods on various tasks, including time series forecasting, text classification, and patch-based image classification, and the experimental results demonstrate a satisfying performance gain by incorporating our RPE methods across many architectures. Our results provide fresh perspectives on designing spiking Transformers to advance their sequential modeling capability, thereby expanding their applicability across various domains.