Real-World Point Tracking with Verifier-Guided Pseudo-Labeling

TL;DR

This paper introduces a verifier-guided pseudo-labeling method for long-term point tracking that improves real-world performance by assessing and selecting reliable predictions for self-training, leading to state-of-the-art results.

Contribution

The paper proposes a verifier meta-model that evaluates tracker predictions to generate high-quality pseudo-labels for effective real-world fine-tuning.

Findings

Achieves state-of-the-art results on four real-world benchmarks.

Requires less data than previous self-training methods.

Significantly improves tracking accuracy in real-world videos.

Abstract

Models for long-term point tracking are typically trained on large synthetic datasets. The performance of these models degrades in real-world videos due to different characteristics and the absence of dense ground-truth annotations. Self-training on unlabeled videos has been explored as a practical solution, but the quality of pseudo-labels strongly depends on the reliability of teacher models, which vary across frames and scenes. In this paper, we address the problem of real-world fine-tuning and introduce verifier, a meta-model that learns to assess the reliability of tracker predictions and guide pseudo-label generation. Given candidate trajectories from multiple pretrained trackers, the verifier evaluates them per frame and selects the most trustworthy predictions, resulting in high-quality pseudo-label trajectories. When applied for fine-tuning, verifier-guided pseudo-labeling substantially improves the quality of supervision and enables data-efficient adaptation to unlabeled videos. Extensive experiments on four real-world benchmarks demonstrate that our approach achieves state-of-the-art results while requiring less data than prior self-training methods. Project page: https://kuis-ai.github.io/track_on_r