Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Models

TL;DR

This paper introduces a fast, latent diffusion-based method for 3D scene generation from 2D images, achieving high-quality results in seconds without requiring depth or masks.

Contribution

The authors develop a novel pipeline combining autoencoders and diffusion models over 3D Gaussian splats, enabling rapid and mask-free 3D scene synthesis from 2D data.

Findings

Generates 3D scenes in as little as 0.2 seconds

Operates without object masks or depth information

Outperforms previous NeRF-based generative models in speed and quality

Abstract

We present a latent diffusion model over 3D scenes, that can be trained using only 2D image data. To achieve this, we first design an autoencoder that maps multi-view images to 3D Gaussian splats, and simultaneously builds a compressed latent representation of these splats. Then, we train a multi-view diffusion model over the latent space to learn an efficient generative model. This pipeline does not require object masks nor depths, and is suitable for complex scenes with arbitrary camera positions. We conduct careful experiments on two large-scale datasets of complex real-world scenes -- MVImgNet and RealEstate10K. We show that our approach enables generating 3D scenes in as little as 0.2 seconds, either from scratch, from a single input view, or from sparse input views. It produces diverse and high-quality results while running an order of magnitude faster than non-latent diffusion models and earlier NeRF-based generative models