Data-Driven Feature Tracking for Event Cameras With and Without Frames

TL;DR

This paper introduces a novel data-driven feature tracking method for event cameras that operates with or without intensity frames, improving robustness and versatility in challenging scenarios.

Contribution

It presents the first data-driven tracker leveraging a frame attention module, capable of hybrid operation with depth estimation for enhanced applications.

Findings

Robust feature tracking achieved with the proposed data-driven method.

Effective hybrid mode combining event data and intensity frames.

Depth information obtained through side-by-side camera setup.

Abstract

Because of their high temporal resolution, increased resilience to motion blur, and very sparse output, event cameras have been shown to be ideal for low-latency and low-bandwidth feature tracking, even in challenging scenarios. Existing feature tracking methods for event cameras are either handcrafted or derived from first principles but require extensive parameter tuning, are sensitive to noise, and do not generalize to different scenarios due to unmodeled effects. To tackle these deficiencies, we introduce the first data-driven feature tracker for event cameras, which leverages low-latency events to track features detected in an intensity frame. We achieve robust performance via a novel frame attention module, which shares information across feature tracks. Our tracker is designed to operate in two distinct configurations: solely with events or in a hybrid mode incorporating both events and frames. The hybrid model offers two setups: an aligned configuration where the event and frame cameras share the same viewpoint, and a hybrid stereo configuration where the event camera and the standard camera are positioned side-by-side. This side-by-side arrangement is particularly valuable as it provides depth information for each feature track, enhancing its utility in applications such as visual odometry and simultaneous localization and mapping.