Emergent Temporal Correspondences from Video Diffusion Transformers

TL;DR

This paper introduces DiffTrack, a framework for analyzing how Diffusion Transformers establish temporal correspondences in videos, revealing key mechanisms and enabling improved zero-shot tracking and generation.

Contribution

DiffTrack provides the first systematic quantitative analysis of temporal correspondence formation in video diffusion transformers, with novel metrics and practical applications.

Findings

Query-key similarities are crucial for temporal matching.

Temporal correspondence becomes stronger during denoising.

DiffTrack achieves state-of-the-art zero-shot point tracking.

Abstract

Recent advancements in video diffusion models based on Diffusion Transformers (DiTs) have achieved remarkable success in generating temporally coherent videos. Yet, a fundamental question persists: how do these models internally establish and represent temporal correspondences across frames? We introduce DiffTrack, the first quantitative analysis framework designed to answer this question. DiffTrack constructs a dataset of prompt-generated video with pseudo ground-truth tracking annotations and proposes novel evaluation metrics to systematically analyze how each component within the full 3D attention mechanism of DiTs (e.g., representations, layers, and timesteps) contributes to establishing temporal correspondences. Our analysis reveals that query-key similarities in specific, but not all, layers play a critical role in temporal matching, and that this matching becomes increasingly prominent during the denoising process. We demonstrate practical applications of DiffTrack in zero-shot point tracking, where it achieves state-of-the-art performance compared to existing vision foundation and self-supervised video models. Further, we extend our findings to motion-enhanced video generation with a novel guidance method that improves temporal consistency of generated videos without additional training. We believe our work offers crucial insights into the inner workings of video DiTs and establishes a foundation for further research and applications leveraging their temporal understanding.