SpikeMOT: Event-based Multi-Object Tracking with Sparse Motion Features

TL;DR

SpikeMOT introduces a novel event-based multi-object tracking method using spiking neural networks to extract sparse features, achieving high accuracy in complex real-world scenarios with occlusions and re-identification.

Contribution

The paper presents SpikeMOT, the first event-based multi-object tracker utilizing spiking neural networks and introduces DSEC-MOT, a large-scale benchmark for real-world evaluation.

Findings

SpikeMOT achieves high tracking accuracy in challenging scenarios.

DSEC-MOT provides detailed annotations for occlusion and re-identification.

Experimental results outperform existing event-based tracking methods.

Abstract

In comparison to conventional RGB cameras, the superior temporal resolution of event cameras allows them to capture rich information between frames, making them prime candidates for object tracking. Yet in practice, despite their theoretical advantages, the body of work on event-based multi-object tracking (MOT) remains in its infancy, especially in real-world settings where events from complex background and camera motion can easily obscure the true target motion. In this work, an event-based multi-object tracker, called SpikeMOT, is presented to address these challenges. SpikeMOT leverages spiking neural networks to extract sparse spatiotemporal features from event streams associated with objects. The resulting spike train representations are used to track the object movement at high frequency, while a simultaneous object detector provides updated spatial information of these objects at an equivalent frame rate. To evaluate the effectiveness of SpikeMOT, we introduce DSEC-MOT, the first large-scale event-based MOT benchmark incorporating fine-grained annotations for objects experiencing severe occlusions, frequent trajectory intersections, and long-term re-identification in real-world contexts. Extensive experiments employing DSEC-MOT and another event-based dataset, named FE240hz, demonstrate SpikeMOT's capability to achieve high tracking accuracy amidst challenging real-world scenarios, advancing the state-of-the-art in event-based multi-object tracking.