Repurposing Video Diffusion Transformers for Robust Point Tracking

TL;DR

This paper demonstrates that pre-trained video Diffusion Transformers can be effectively adapted for robust point tracking, outperforming existing methods especially under challenging conditions.

Contribution

The paper introduces DiTracker, a novel approach that leverages video DiT features with minimal tuning, achieving state-of-the-art results in point tracking benchmarks.

Findings

DiTracker outperforms existing methods on ITTO benchmark.

Video DiT features provide strong point tracking capabilities.

The approach is efficient with smaller training batch sizes.

Abstract

Point tracking aims to localize corresponding points across video frames, serving as a fundamental task for 4D reconstruction, robotics, and video editing. Existing methods commonly rely on shallow convolutional backbones such as ResNet that process frames independently, lacking temporal coherence and producing unreliable matching costs under challenging conditions. Through systematic analysis, we find that video Diffusion Transformers (DiTs), pre-trained on large-scale real-world videos with spatio-temporal attention, inherently exhibit strong point tracking capability and robustly handle dynamic motions and frequent occlusions. We propose DiTracker, which adapts video DiTs through: (1) query-key attention matching, (2) lightweight LoRA tuning, and (3) cost fusion with a ResNet backbone. Despite training with 8 times smaller batch size, DiTracker achieves state-of-the-art performance on challenging ITTO benchmark and matches or outperforms state-of-the-art models on TAP-Vid benchmarks. Our work validates video DiT features as an effective and efficient foundation for point tracking.