HDI-Former: Hybrid Dynamic Interaction ANN-SNN Transformer for Object Detection Using Frames and Events

TL;DR

HDI-Former introduces a hybrid ANN-SNN transformer architecture that effectively combines frame and event data for high-accuracy, energy-efficient object detection, utilizing novel attention and interaction mechanisms.

Contribution

It is the first to design a directly trained hybrid ANN-SNN architecture for object detection using frames and events, enhancing cross-modality interaction and energy efficiency.

Findings

Outperforms eleven state-of-the-art methods significantly.

SNN branch achieves comparable accuracy to ANN with 10.57× less energy.

Effective modeling of temporal cues from event streams.

Abstract

Combining the complementary benefits of frames and events has been widely used for object detection in challenging scenarios. However, most object detection methods use two independent Artificial Neural Network (ANN) branches, limiting cross-modality information interaction across the two visual streams and encountering challenges in extracting temporal cues from event streams with low power consumption. To address these challenges, we propose HDI-Former, a Hybrid Dynamic Interaction ANN-SNN Transformer, marking the first trial to design a directly trained hybrid ANN-SNN architecture for high-accuracy and energy-efficient object detection using frames and events. Technically, we first present a novel semantic-enhanced self-attention mechanism that strengthens the correlation between image encoding tokens within the ANN Transformer branch for better performance. Then, we design a Spiking Swin Transformer branch to model temporal cues from event streams with low power consumption. Finally, we propose a bio-inspired dynamic interaction mechanism between ANN and SNN sub-networks for cross-modality information interaction. The results demonstrate that our HDI-Former outperforms eleven state-of-the-art methods and our four baselines by a large margin. Our SNN branch also shows comparable performance to the ANN with the same architecture while consuming 10.57$\times$ less energy on the DSEC-Detection dataset. Our open-source code is available in the supplementary material.