DreamGaussian4D: Generative 4D Gaussian Splatting

TL;DR

DreamGaussian4D introduces an efficient 4D generation framework that combines Gaussian Splatting with explicit spatial transformations and video priors, significantly reducing optimization time and enabling controllable, high-quality animated 4D content.

Contribution

It presents a novel framework integrating static and dynamic Gaussian Splatting with video-based refinement, achieving fast, controllable, and high-quality 4D content generation.

Findings

Optimization time reduced from hours to minutes

Enables visual control of 3D motion

Produces realistic animated meshes for 3D rendering

Abstract

4D content generation has achieved remarkable progress recently. However, existing methods suffer from long optimization times, a lack of motion controllability, and a low quality of details. In this paper, we introduce DreamGaussian4D (DG4D), an efficient 4D generation framework that builds on Gaussian Splatting (GS). Our key insight is that combining explicit modeling of spatial transformations with static GS makes an efficient and powerful representation for 4D generation. Moreover, video generation methods have the potential to offer valuable spatial-temporal priors, enhancing the high-quality 4D generation. Specifically, we propose an integral framework with two major modules: 1) Image-to-4D GS - we initially generate static GS with DreamGaussianHD, followed by HexPlane-based dynamic generation with Gaussian deformation; and 2) Video-to-Video Texture Refinement - we refine the generated UV-space texture maps and meanwhile enhance their temporal consistency by utilizing a pre-trained image-to-video diffusion model. Notably, DG4D reduces the optimization time from several hours to just a few minutes, allows the generated 3D motion to be visually controlled, and produces animated meshes that can be realistically rendered in 3D engines.