Tracking-Guided 4D Generation: Foundation-Tracker Motion Priors for 3D Model Animation

TL;DR

This paper introduces Track4DGen, a novel framework that integrates explicit tracker-derived motion priors into a two-stage process for generating temporally coherent 4D dynamic objects from sparse inputs, improving stability and fidelity.

Contribution

The paper presents a new two-stage method combining multi-view diffusion and point tracking to enhance 4D object generation with explicit motion priors, surpassing existing baselines.

Findings

Outperforms baselines on multi-view video and 4D generation benchmarks.

Produces temporally stable and text-editable 4D assets.

Curates a new dataset, Sketchfab28, for benchmarking 4D generation.

Abstract

Generating dynamic 4D objects from sparse inputs is difficult because it demands joint preservation of appearance and motion coherence across views and time while suppressing artifacts and temporal drift. We hypothesize that the view discrepancy arises from supervision limited to pixel- or latent-space video-diffusion losses, which lack explicitly temporally aware, feature-level tracking guidance. We present \emph{Track4DGen}, a two-stage framework that couples a multi-view video diffusion model with a foundation point tracker and a hybrid 4D Gaussian Splatting (4D-GS) reconstructor. The central idea is to explicitly inject tracker-derived motion priors into intermediate feature representations for both multi-view video generation and 4D-GS. In Stage One, we enforce dense, feature-level point correspondences inside the diffusion generator, producing temporally consistent features that curb appearance drift and enhance cross-view coherence. In Stage Two, we reconstruct a dynamic 4D-GS using a hybrid motion encoding that concatenates co-located diffusion features (carrying Stage-One tracking priors) with Hex-plane features, and augment them with 4D Spherical Harmonics for higher-fidelity dynamics modeling. \emph{Track4DGen} surpasses baselines on both multi-view video generation and 4D generation benchmarks, yielding temporally stable, text-editable 4D assets. Lastly, we curate \emph{Sketchfab28}, a high-quality dataset for benchmarking object-centric 4D generation and fostering future research.