SpatialTrackerV2: 3D Point Tracking Made Easy

TL;DR

SpatialTrackerV2 introduces a unified, end-to-end 3D point tracking method for monocular videos that jointly learns geometry and motion, achieving superior accuracy and speed over existing methods.

Contribution

It unifies 3D point tracking, depth, and pose estimation into a differentiable architecture, enabling scalable training and improved performance.

Findings

Outperforms existing 3D tracking methods by 30%

Matches accuracy of top dynamic 3D reconstruction methods

Runs 50 times faster than comparable approaches

Abstract

We present SpatialTrackerV2, a feed-forward 3D point tracking method for monocular videos. Going beyond modular pipelines built on off-the-shelf components for 3D tracking, our approach unifies the intrinsic connections between point tracking, monocular depth, and camera pose estimation into a high-performing and feedforward 3D point tracker. It decomposes world-space 3D motion into scene geometry, camera ego-motion, and pixel-wise object motion, with a fully differentiable and end-to-end architecture, allowing scalable training across a wide range of datasets, including synthetic sequences, posed RGB-D videos, and unlabeled in-the-wild footage. By learning geometry and motion jointly from such heterogeneous data, SpatialTrackerV2 outperforms existing 3D tracking methods by 30%, and matches the accuracy of leading dynamic 3D reconstruction approaches while running 50$\times$ faster.