RealisticDreamer: Guidance Score Distillation for Few-shot Gaussian Splatting

TL;DR

RealisticDreamer introduces Guidance Score Distillation, leveraging pretrained Video Diffusion Models to improve 3D Gaussian Splatting from sparse views by incorporating depth and semantic guidance, leading to better multi-view consistency.

Contribution

The paper proposes Guidance Score Distillation, a novel framework that enhances 3D Gaussian Splatting with multi-view priors from VDMs and unified guidance to prevent overfitting in sparse view scenarios.

Findings

Outperforms existing methods on multiple datasets.

Improves multi-view consistency in sparse view settings.

Effectively incorporates depth and semantic guidance.

Abstract

3D Gaussian Splatting (3DGS) has recently gained great attention in the 3D scene representation for its high-quality real-time rendering capabilities. However, when the input comprises sparse training views, 3DGS is prone to overfitting, primarily due to the lack of intermediate-view supervision. Inspired by the recent success of Video Diffusion Models (VDM), we propose a framework called Guidance Score Distillation (GSD) to extract the rich multi-view consistency priors from pretrained VDMs. Building on the insights from Score Distillation Sampling (SDS), GSD supervises rendered images from multiple neighboring views, guiding the Gaussian splatting representation towards the generative direction of VDM. However, the generative direction often involves object motion and random camera trajectories, making it challenging for direct supervision in the optimization process. To address this problem, we introduce an unified guidance form to correct the noise prediction result of VDM. Specifically, we incorporate both a depth warp guidance based on real depth maps and a guidance based on semantic image features, ensuring that the score update direction from VDM aligns with the correct camera pose and accurate geometry. Experimental results show that our method outperforms existing approaches across multiple datasets.