DSplats: 3D Generation by Denoising Splats-Based Multiview Diffusion Models

TL;DR

DSplats introduces a novel 3D generation method that combines multiview diffusion models with Gaussian splats, enabling high-quality, consistent 3D asset creation from limited input images by leveraging pretrained diffusion priors.

Contribution

The paper presents DSplats, integrating 2D diffusion models with Gaussian splats for explicit 3D priors, improving 3D reconstruction quality and view consistency from sparse images.

Findings

Achieves PSNR of 20.38 on Google Scanned Objects dataset

Attains SSIM of 0.842, LPIPS of 0.109

Produces spatially consistent, high-quality 3D outputs

Abstract

Generating high-quality 3D content requires models capable of learning robust distributions of complex scenes and the real-world objects within them. Recent Gaussian-based 3D reconstruction techniques have achieved impressive results in recovering high-fidelity 3D assets from sparse input images by predicting 3D Gaussians in a feed-forward manner. However, these techniques often lack the extensive priors and expressiveness offered by Diffusion Models. On the other hand, 2D Diffusion Models, which have been successfully applied to denoise multiview images, show potential for generating a wide range of photorealistic 3D outputs but still fall short on explicit 3D priors and consistency. In this work, we aim to bridge these two approaches by introducing DSplats, a novel method that directly denoises multiview images using Gaussian Splat-based Reconstructors to produce a diverse array of realistic 3D assets. To harness the extensive priors of 2D Diffusion Models, we incorporate a pretrained Latent Diffusion Model into the reconstructor backbone to predict a set of 3D Gaussians. Additionally, the explicit 3D representation embedded in the denoising network provides a strong inductive bias, ensuring geometrically consistent novel view generation. Our qualitative and quantitative experiments demonstrate that DSplats not only produces high-quality, spatially consistent outputs, but also sets a new standard in single-image to 3D reconstruction. When evaluated on the Google Scanned Objects dataset, DSplats achieves a PSNR of 20.38, an SSIM of 0.842, and an LPIPS of 0.109.