EgoPoints: Advancing Point Tracking for Egocentric Videos

TL;DR

EgoPoints introduces a new benchmark and dataset for point tracking in egocentric videos, emphasizing challenging scenarios like out-of-view points and re-identification, and demonstrates improved tracking performance with fine-tuning.

Contribution

The paper presents EgoPoints, a comprehensive benchmark with annotated sequences and evaluation metrics, along with a semi-real sequence generation pipeline for training and testing egocentric point tracking models.

Findings

Fine-tuning on EgoPoints sequences improves CoTracker accuracy by 2.7 percentage points.

ReID accuracy increases by 2.4 points after fine-tuning.

The benchmark includes 9x more out-of-view points and 59x more re-identification points than previous datasets.

Abstract

We introduce EgoPoints, a benchmark for point tracking in egocentric videos. We annotate 4.7K challenging tracks in egocentric sequences. Compared to the popular TAP-Vid-DAVIS evaluation benchmark, we include 9x more points that go out-of-view and 59x more points that require re-identification (ReID) after returning to view. To measure the performance of models on these challenging points, we introduce evaluation metrics that specifically monitor tracking performance on points in-view, out-of-view, and points that require re-identification. We then propose a pipeline to create semi-real sequences, with automatic ground truth. We generate 11K such sequences by combining dynamic Kubric objects with scene points from EPIC Fields. When fine-tuning point tracking methods on these sequences and evaluating on our annotated EgoPoints sequences, we improve CoTracker across all metrics, including the tracking accuracy $\delta^\star_{\text{avg}}$ by 2.7 percentage points and accuracy on ReID sequences (ReID$\delta_{\text{avg}}$) by 2.4 points. We also improve $\delta^\star_{\text{avg}}$ and ReID$\delta_{\text{avg}}$ of PIPs++ by 0.3 and 2.8 respectively.