Tracking Any Point with Frame-Event Fusion Network at High Frame Rate

TL;DR

This paper introduces FE-TAP, a novel image-event fusion point tracker that combines image frames and event data to achieve high frame rate and robustness in point tracking, especially in high-speed scenarios.

Contribution

The paper presents FE-TAP with the EvoFusion module and transformer-based refinement, enabling effective multi-modal fusion and improved tracking performance over state-of-the-art methods.

Findings

Outperforms existing methods in high-speed point tracking.

Improves feature age by 24% on EDS datasets.

Demonstrates robustness in real driving scenarios.

Abstract

Tracking any point based on image frames is constrained by frame rates, leading to instability in high-speed scenarios and limited generalization in real-world applications. To overcome these limitations, we propose an image-event fusion point tracker, FE-TAP, which combines the contextual information from image frames with the high temporal resolution of events, achieving high frame rate and robust point tracking under various challenging conditions. Specifically, we designed an Evolution Fusion module (EvoFusion) to model the image generation process guided by events. This module can effectively integrate valuable information from both modalities operating at different frequencies. To achieve smoother point trajectories, we employed a transformer-based refinement strategy that updates the point's trajectories and features iteratively. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches, particularly improving expected feature age by 24$\%$ on EDS datasets. Finally, we qualitatively validated the robustness of our algorithm in real driving scenarios using our custom-designed high-resolution image-event synchronization device. Our source code will be released at https://github.com/ljx1002/FE-TAP.