ETAP: Event-based Tracking of Any Point

TL;DR

ETAP introduces a novel event camera-based method for tracking any point that leverages high temporal resolution and global context to achieve robust high-speed tracking, outperforming existing approaches significantly.

Contribution

This work is the first to develop an event camera-based TAP method, incorporating a feature-alignment loss for motion-robust features and demonstrating strong generalization and performance improvements.

Findings

136% better on average Jaccard metric than baselines

20% improvement on feature tracking benchmark over previous event-only method

Surpasses previous best events-and-frames method by 4.1%

Abstract

Tracking any point (TAP) recently shifted the motion estimation paradigm from focusing on individual salient points with local templates to tracking arbitrary points with global image contexts. However, while research has mostly focused on driving the accuracy of models in nominal settings, addressing scenarios with difficult lighting conditions and high-speed motions remains out of reach due to the limitations of the sensor. This work addresses this challenge with the first event camera-based TAP method. It leverages the high temporal resolution and high dynamic range of event cameras for robust high-speed tracking, and the global contexts in TAP methods to handle asynchronous and sparse event measurements. We further extend the TAP framework to handle event feature variations induced by motion -- thereby addressing an open challenge in purely event-based tracking -- with a novel feature-alignment loss which ensures the learning of motion-robust features. Our method is trained with data from a new data generation pipeline and systematically ablated across all design decisions. Our method shows strong cross-dataset generalization and performs 136% better on the average Jaccard metric than the baselines. Moreover, on an established feature tracking benchmark, it achieves a 20% improvement over the previous best event-only method and even surpasses the previous best events-and-frames method by 4.1%. Our code is available at https://github.com/tub-rip/ETAP