Frame-Event Alignment and Fusion Network for High Frame Rate Tracking

TL;DR

This paper introduces a novel end-to-end network that combines conventional RGB frames and event-based camera data to enable high frame rate object tracking, significantly surpassing existing methods in challenging conditions.

Contribution

It proposes a multi-modality alignment and fusion network that effectively integrates frame and event data for high frame rate tracking, a novel approach in this domain.

Findings

Outperforms state-of-the-art trackers in high frame rate scenarios

Achieves tracking up to 240Hz on the FE240hz dataset

Effectively handles challenging conditions with multi-modality fusion

Abstract

Most existing RGB-based trackers target low frame rate benchmarks of around 30 frames per second. This setting restricts the tracker's functionality in the real world, especially for fast motion. Event-based cameras as bioinspired sensors provide considerable potential for high frame rate tracking due to their high temporal resolution. However, event-based cameras cannot offer fine-grained texture information like conventional cameras. This unique complementarity motivates us to combine conventional frames and events for high frame rate object tracking under various challenging conditions. Inthispaper, we propose an end-to-end network consisting of multi-modality alignment and fusion modules to effectively combine meaningful information from both modalities at different measurement rates. The alignment module is responsible for cross-style and cross-frame-rate alignment between frame and event modalities under the guidance of the moving cues furnished by events. While the fusion module is accountable for emphasizing valuable features and suppressing noise information by the mutual complement between the two modalities. Extensive experiments show that the proposed approach outperforms state-of-the-art trackers by a significant margin in high frame rate tracking. With the FE240hz dataset, our approach achieves high frame rate tracking up to 240Hz.