BlinkTrack: Feature Tracking over 80 FPS via Events and Images

TL;DR

BlinkTrack is a novel high-frequency feature tracking framework that combines event camera data with images, extending Kalman filters into a learning-based approach to achieve over 80 FPS in challenging conditions.

Contribution

The paper introduces a differentiable Kalman filter-based framework that fuses event and image data for improved feature tracking, along with new datasets for evaluation.

Findings

Achieves over 80 FPS with multi-modality data

Outperforms existing methods in accuracy and speed

Effectively fuses asynchronous event and image data

Abstract

Event cameras, known for their high temporal resolution and ability to capture asynchronous changes, have gained significant attention for their potential in feature tracking, especially in challenging conditions. However, event cameras lack the fine-grained texture information that conventional cameras provide, leading to error accumulation in tracking. To address this, we propose a novel framework, BlinkTrack, which integrates event data with grayscale images for high-frequency feature tracking. Our method extends the traditional Kalman filter into a learning-based framework, utilizing differentiable Kalman filters in both event and image branches. This approach improves single-modality tracking and effectively solves the data association and fusion from asynchronous event and image data. We also introduce new synthetic and augmented datasets to better evaluate our model. Experimental results indicate that BlinkTrack significantly outperforms existing methods, exceeding 80 FPS with multi-modality data and 100 FPS with preprocessed event data. Codes and dataset are available at https://github.com/ColieShen/BlinkTrack.