CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos

TL;DR

CoTracker3 introduces a simplified semi-supervised point tracking model trained on real videos with pseudo-labels, outperforming previous methods trained on synthetic data, and effectively tracking occluded points.

Contribution

It presents a new semi-supervised training recipe and a simplified model architecture that leverages real videos with pseudo-labels, reducing data requirements and improving performance.

Findings

Achieves better tracking results with 1,000x less data

Effectively tracks occluded and visible points

Simplifies previous tracking architectures

Abstract

Most state-of-the-art point trackers are trained on synthetic data due to the difficulty of annotating real videos for this task. However, this can result in suboptimal performance due to the statistical gap between synthetic and real videos. In order to understand these issues better, we introduce CoTracker3, comprising a new tracking model and a new semi-supervised training recipe. This allows real videos without annotations to be used during training by generating pseudo-labels using off-the-shelf teachers. The new model eliminates or simplifies components from previous trackers, resulting in a simpler and often smaller architecture. This training scheme is much simpler than prior work and achieves better results using 1,000 times less data. We further study the scaling behaviour to understand the impact of using more real unsupervised data in point tracking. The model is available in online and offline variants and reliably tracks visible and occluded points.