ScalingGaussian: Enhancing 3D Content Creation with Generative Gaussian Splatting

TL;DR

ScalingGaussian is a novel framework that combines 3D and 2D diffusion models to generate detailed, high-quality 3D assets with strong geometric consistency, suitable for accessible content creation.

Contribution

It introduces a new method integrating 3D and 2D diffusion models with Gaussian noise refinement and texture optimization for improved 3D asset generation.

Findings

Produces high-quality 3D assets with detailed textures and consistent geometry

Addresses sparse point cloud issues in 3D diffusion

Efficiently generates 3D models suitable for various applications

Abstract

The creation of high-quality 3D assets is paramount for applications in digital heritage preservation, entertainment, and robotics. Traditionally, this process necessitates skilled professionals and specialized software for the modeling, texturing, and rendering of 3D objects. However, the rising demand for 3D assets in gaming and virtual reality (VR) has led to the creation of accessible image-to-3D technologies, allowing non-professionals to produce 3D content and decreasing dependence on expert input. Existing methods for 3D content generation struggle to simultaneously achieve detailed textures and strong geometric consistency. We introduce a novel 3D content creation framework, ScalingGaussian, which combines 3D and 2D diffusion models to achieve detailed textures and geometric consistency in generated 3D assets. Initially, a 3D diffusion model generates point clouds, which are then densified through a process of selecting local regions, introducing Gaussian noise, followed by using local density-weighted selection. To refine the 3D gaussians, we utilize a 2D diffusion model with Score Distillation Sampling (SDS) loss, guiding the 3D Gaussians to clone and split. Finally, the 3D Gaussians are converted into meshes, and the surface textures are optimized using Mean Square Error(MSE) and Gradient Profile Prior(GPP) losses. Our method addresses the common issue of sparse point clouds in 3D diffusion, resulting in improved geometric structure and detailed textures. Experiments on image-to-3D tasks demonstrate that our approach efficiently generates high-quality 3D assets.