Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding

TL;DR

Motion4D introduces a 4D Gaussian Splatting framework that integrates 2D priors into a unified 3D-consistent scene understanding model, significantly improving motion and semantic coherence in dynamic 3D environments.

Contribution

The paper presents a novel 4D Gaussian Splatting approach with iterative optimization, dynamic motion priors, and semantic refinement, addressing 3D consistency issues in scene understanding from monocular videos.

Findings

Outperforms existing 2D and 3D methods in scene understanding tasks.

Achieves superior 3D motion and semantic consistency.

Enhances dynamic scene analysis with improved accuracy.

Abstract

Recent advancements in foundation models for 2D vision have substantially improved the analysis of dynamic scenes from monocular videos. However, despite their strong generalization capabilities, these models often lack 3D consistency, a fundamental requirement for understanding scene geometry and motion, thereby causing severe spatial misalignment and temporal flickering in complex 3D environments. In this paper, we present Motion4D, a novel framework that addresses these challenges by integrating 2D priors from foundation models into a unified 4D Gaussian Splatting representation. Our method features a two-part iterative optimization framework: 1) Sequential optimization, which updates motion and semantic fields in consecutive stages to maintain local consistency, and 2) Global optimization, which jointly refines all attributes for long-term coherence. To enhance motion accuracy, we introduce a 3D confidence map that dynamically adjusts the motion priors, and an adaptive resampling process that inserts new Gaussians into under-represented regions based on per-pixel RGB and semantic errors. Furthermore, we enhance semantic coherence through an iterative refinement process that resolves semantic inconsistencies by alternately optimizing the semantic fields and updating prompts of SAM2. Extensive evaluations demonstrate that our Motion4D significantly outperforms both 2D foundation models and existing 3D-based approaches across diverse scene understanding tasks, including point-based tracking, video object segmentation, and novel view synthesis. Our code is available at https://hrzhou2.github.io/motion4d-web/.