VGGT4D: Mining Motion Cues in Visual Geometry Transformers for 4D Scene Reconstruction

TL;DR

VGGT4D is a training-free framework that enhances 4D scene reconstruction by mining and amplifying dynamic cues from a 3D foundation model, enabling robust segmentation and pose estimation in dynamic scenes.

Contribution

It introduces a novel method to mine dynamic cues from VGGT's attention layers and integrates them into inference, eliminating the need for external priors or fine-tuning.

Findings

Outperforms existing methods in dynamic segmentation and reconstruction

Supports single-pass inference on sequences over 500 frames

Effectively disentangles static and dynamic scene elements

Abstract

Reconstructing dynamic 4D scenes is challenging, as it requires robust disentanglement of dynamic objects from the static background. While 3D foundation models like VGGT provide accurate 3D geometry, their performance drops markedly when moving objects dominate. Existing 4D approaches often rely on external priors, heavy post-optimization, or require fine-tuning on 4D datasets. In this paper, we propose VGGT4D, a training-free framework that extends the 3D foundation model VGGT for robust 4D scene reconstruction. Our approach is motivated by the key finding that VGGT's global attention layers already implicitly encode rich, layer-wise dynamic cues. To obtain masks that decouple static and dynamic elements, we mine and amplify global dynamic cues via gram similarity and aggregate them across a temporal window. To further sharpen mask boundaries, we introduce a refinement strategy driven by projection gradient. We then integrate these precise masks into VGGT's early-stage inference, effectively mitigating motion interference in both pose estimation and geometric reconstruction. Across six datasets, our method achieves superior performance in dynamic object segmentation, camera pose estimation, and dense reconstruction. It also supports single-pass inference on sequences longer than 500 frames.