GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling

TL;DR

GaussianCube is a novel explicit 3D radiance representation that uses a structured grid of Gaussians, enabling efficient and high-quality 3D generative modeling with fewer parameters and easy integration with diffusion methods.

Contribution

We propose GaussianCube, a structured and explicit Gaussian-based radiance representation that simplifies 3D generative modeling and achieves state-of-the-art results with significantly fewer parameters.

Findings

Achieves state-of-the-art 3D generation quality.

Uses orders of magnitude fewer parameters than previous methods.

Demonstrates versatility across multiple 3D generation tasks.

Abstract

We introduce a radiance representation that is both structured and fully explicit and thus greatly facilitates 3D generative modeling. Existing radiance representations either require an implicit feature decoder, which significantly degrades the modeling power of the representation, or are spatially unstructured, making them difficult to integrate with mainstream 3D diffusion methods. We derive GaussianCube by first using a novel densification-constrained Gaussian fitting algorithm, which yields high-accuracy fitting using a fixed number of free Gaussians, and then rearranging these Gaussians into a predefined voxel grid via Optimal Transport. Since GaussianCube is a structured grid representation, it allows us to use standard 3D U-Net as our backbone in diffusion modeling without elaborate designs. More importantly, the high-accuracy fitting of the Gaussians allows us to achieve a high-quality representation with orders of magnitude fewer parameters than previous structured representations for comparable quality, ranging from one to two orders of magnitude. The compactness of GaussianCube greatly eases the difficulty of 3D generative modeling. Extensive experiments conducted on unconditional and class-conditioned object generation, digital avatar creation, and text-to-3D synthesis all show that our model achieves state-of-the-art generation results both qualitatively and quantitatively, underscoring the potential of GaussianCube as a highly accurate and versatile radiance representation for 3D generative modeling. Project page: https://gaussiancube.github.io/.