AnthroTAP: Learning Point Tracking with Real-World Motion

TL;DR

AnthroTAP introduces an automated pipeline that generates large-scale pseudo-labeled point tracking data from real human videos, improving real-world generalization of tracking models.

Contribution

It presents a novel method to produce real-world training data for point tracking by fitting SMPL models to human videos, reducing reliance on expensive manual annotations.

Findings

Model trained on AnthroTAP surpasses state-of-the-art on TAP-Vid.

Outperforms recent self-training methods with less training time.

Structured human motion is an effective supervision source.

Abstract

Point tracking models often struggle to generalize to real-world videos because large-scale training data is predominantly synthetic$\unicode{x2014}$the only source currently feasible to produce at scale. Collecting real-world annotations, however, is prohibitively expensive, as it requires tracking hundreds of points across frames. We introduce \textbf{AnthroTAP}, an automated pipeline that generates large-scale pseudo-labeled point tracking data from real human motion videos. Leveraging the structured complexity of human movement$\unicode{x2014}$non-rigid deformations, articulated motion, and frequent occlusions$\unicode{x2014}$AnthroTAP fits Skinned Multi-Person Linear (SMPL) models to detected humans, projects mesh vertices onto image planes, resolves occlusions via ray-casting, and filters unreliable tracks using optical flow consistency. A model trained on the AnthroTAP dataset achieves state-of-the-art performance on TAP-Vid, a challenging general-domain benchmark for tracking any point on diverse rigid and non-rigid objects (e.g., humans, animals, robots, and vehicles). Our approach outperforms recent self-training methods trained on vastly larger real datasets, while requiring only one day of training on 4 GPUs. AnthroTAP shows that structured human motion offers a scalable and effective source of real-world supervision for point tracking.