VDEGaussian: Video Diffusion Enhanced 4D Gaussian Splatting for Dynamic Urban Scenes Modeling

TL;DR

This paper introduces VDEGaussian, a novel framework that enhances 4D Gaussian Splatting for dynamic urban scene modeling by integrating video diffusion priors, improving fast-moving object reconstruction and view synthesis quality.

Contribution

It proposes a test-time adapted video diffusion-based approach with pose optimization and uncertainty distillation to improve dynamic scene modeling.

Findings

Achieves approximately 2 dB PSNR gain over baseline methods.

Effectively models fast-moving objects in urban scenes.

Enhances novel view synthesis quality.

Abstract

Dynamic urban scene modeling is a rapidly evolving area with broad applications. While current approaches leveraging neural radiance fields or Gaussian Splatting have achieved fine-grained reconstruction and high-fidelity novel view synthesis, they still face significant limitations. These often stem from a dependence on pre-calibrated object tracks or difficulties in accurately modeling fast-moving objects from undersampled capture, particularly due to challenges in handling temporal discontinuities. To overcome these issues, we propose a novel video diffusion-enhanced 4D Gaussian Splatting framework. Our key insight is to distill robust, temporally consistent priors from a test-time adapted video diffusion model. To ensure precise pose alignment and effective integration of this denoised content, we introduce two core innovations: a joint timestamp optimization strategy that refines interpolated frame poses, and an uncertainty distillation method that adaptively extracts target content while preserving well-reconstructed regions. Extensive experiments demonstrate that our method significantly enhances dynamic modeling, especially for fast-moving objects, achieving an approximate PSNR gain of 2 dB for novel view synthesis over baseline approaches.