3DGen: Triplane Latent Diffusion for Textured Mesh Generation

TL;DR

3DGen introduces a novel two-step pipeline combining a triplane VAE and a diffusion model to generate high-quality textured 3D meshes efficiently, advancing 3D generative modeling.

Contribution

The paper presents the first architecture capable of conditional and unconditional textured mesh generation using a triplane VAE and diffusion model, achieving high quality and diversity.

Findings

Outperforms previous methods in mesh quality and texture generation.

Generates high-quality textured meshes in seconds on a single GPU.

Scales effectively to large datasets for improved diversity.

Abstract

Latent diffusion models for image generation have crossed a quality threshold which enabled them to achieve mass adoption. Recently, a series of works have made advancements towards replicating this success in the 3D domain, introducing techniques such as point cloud VAE, triplane representation, neural implicit surfaces and differentiable rendering based training. We take another step along this direction, combining these developments in a two-step pipeline consisting of 1) a triplane VAE which can learn latent representations of textured meshes and 2) a conditional diffusion model which generates the triplane features. For the first time this architecture allows conditional and unconditional generation of high quality textured or untextured 3D meshes across multiple diverse categories in a few seconds on a single GPU. It outperforms previous work substantially on image-conditioned and unconditional generation on mesh quality as well as texture generation. Furthermore, we demonstrate the scalability of our model to large datasets for increased quality and diversity. We will release our code and trained models.