ProTracker: Probabilistic Integration for Robust and Accurate Point Tracking

TL;DR

ProTracker introduces a probabilistic framework that combines local optical flow and global heatmaps to achieve accurate, robust long-term dense point tracking in videos, outperforming existing methods.

Contribution

It presents a novel probabilistic approach that integrates local and global cues for improved long-term point tracking in videos.

Findings

Achieves state-of-the-art results among optimization-based methods.

Surpasses supervised feed-forward methods on multiple benchmarks.

Effectively handles occlusions and scene changes.

Abstract

We propose ProTracker, a novel framework for accurate and robust long-term dense tracking of arbitrary points in videos. Previous methods relying on global cost volumes effectively handle large occlusions and scene changes but lack precision and temporal awareness. In contrast, local iteration-based methods accurately track smoothly transforming scenes but face challenges with occlusions and drift. To address these issues, we propose a probabilistic framework that marries the strengths of both paradigms by leveraging local optical flow for predictions and refined global heatmaps for observations. This design effectively combines global semantic information with temporally aware low-level features, enabling precise and robust long-term tracking of arbitrary points in videos. Extensive experiments demonstrate that ProTracker attains state-of-the-art performance among optimization-based approaches and surpasses supervised feed-forward methods on multiple benchmarks. The code and model will be released after publication.