Long-Lived Accurate Keypoints in Event Streams

TL;DR

This paper introduces a new end-to-end method for detecting and tracking keypoints in event streams, achieving significantly longer and more accurate keypoint tracks by predicting trajectories over time and using stable training labels.

Contribution

The paper presents a novel training procedure for stable keypoint labels and a trajectory-based prediction approach, improving keypoint tracking duration and accuracy in event streams.

Findings

Keypoint tracks are three times longer than previous methods.

Keypoint localization accuracy is nearly doubled.

The approach generalizes well to different event-based datasets.

Abstract

We present a novel end-to-end approach to keypoint detection and tracking in an event stream that provides better precision and much longer keypoint tracks than previous methods. This is made possible by two contributions working together. First, we propose a simple procedure to generate stable keypoint labels, which we use to train a recurrent architecture. This training data results in detections that are very consistent over time. Moreover, we observe that previous methods for keypoint detection work on a representation (such as the time surface) that integrates events over a period of time. Since this integration is required, we claim it is better to predict the keypoints' trajectories for the time period rather than single locations, as done in previous approaches. We predict these trajectories in the form of a series of heatmaps for the integration time period. This improves the keypoint localization. Our architecture can also be kept very simple, which results in very fast inference times. We demonstrate our approach on the HVGA ATIS Corner dataset as well as "The Event-Camera Dataset and Simulator" dataset, and show it results in keypoint tracks that are three times longer and nearly twice as accurate as the best previous state-of-the-art methods. We believe our approach can be generalized to other event-based camera problems, and we release our source code to encourage other authors to explore it.