GaussianGrow: Geometry-aware Gaussian Growing from 3D Point Clouds with Text Guidance

TL;DR

GaussianGrow introduces a geometry-aware, text-guided method for generating 3D Gaussians from point clouds, improving geometric accuracy and rendering quality through multi-view diffusion and iterative inpainting.

Contribution

It proposes a novel approach that learns to grow 3D Gaussians from point clouds using text guidance, addressing geometric inaccuracies of prior methods.

Findings

Effective in generating 3D Gaussians from synthetic and real point clouds.

Improves geometric accuracy and visual consistency in 3D Gaussian generation.

Demonstrates superior rendering quality compared to existing methods.

Abstract

3D Gaussian Splatting has demonstrated superior performance in rendering efficiency and quality, yet the generation of 3D Gaussians still remains a challenge without proper geometric priors. Existing methods have explored predicting point maps as geometric references for inferring Gaussian primitives, while the unreliable estimated geometries may lead to poor generations. In this work, we introduce GaussianGrow, a novel approach that generates 3D Gaussians by learning to grow them from easily accessible 3D point clouds, naturally enforcing geometric accuracy in Gaussian generation. Specifically, we design a text-guided Gaussian growing scheme that leverages a multi-view diffusion model to synthesize consistent appearances from input point clouds for supervision. To mitigate artifacts caused by fusing neighboring views, we constrain novel views generated at non-preset camera poses identified in overlapping regions across different views. For completing the hard-to-observe regions, we propose to iteratively detect the camera pose by observing the largest un-grown regions in point clouds and inpainting them by inpainting the rendered view with a pretrained 2D diffusion model. The process continues until complete Gaussians are generated. We extensively evaluate GaussianGrow on text-guided Gaussian generation from synthetic and even real-scanned point clouds. Project Page: https://weiqi-zhang.github.io/GaussianGrow