Generative 4D Scene Gaussian Splatting with Object View-Synthesis Priors

TL;DR

GenMOJO is a novel method that generates dynamic 4D scenes from monocular videos by decomposing scenes into objects, optimizing deformable Gaussians, and leveraging generative priors for realistic view synthesis.

Contribution

It introduces a scene decomposition approach with object-wise Gaussian optimization and integrates generative priors for improved 4D scene reconstruction and view synthesis.

Findings

Outperforms existing methods in realistic scene rendering

Produces more accurate 2D and 3D point tracks

Generates highly realistic novel views

Abstract

We tackle the challenge of generating dynamic 4D scenes from monocular, multi-object videos with heavy occlusions, and introduce GenMOJO, a novel approach that integrates rendering-based deformable 3D Gaussian optimization with generative priors for view synthesis. While existing models perform well on novel view synthesis for isolated objects, they struggle to generalize to complex, cluttered scenes. To address this, GenMOJO decomposes the scene into individual objects, optimizing a differentiable set of deformable Gaussians per object. This object-wise decomposition allows leveraging object-centric diffusion models to infer unobserved regions in novel viewpoints. It performs joint Gaussian splatting to render the full scene, capturing cross-object occlusions, and enabling occlusion-aware supervision. To bridge the gap between object-centric priors and the global frame-centric coordinate system of videos, GenMOJO uses differentiable transformations that align generative and rendering constraints within a unified framework. The resulting model generates 4D object reconstructions over space and time, and produces accurate 2D and 3D point tracks from monocular input. Quantitative evaluations and perceptual human studies confirm that GenMOJO generates more realistic novel views of scenes and produces more accurate point tracks compared to existing approaches.