Shap-E: Generating Conditional 3D Implicit Functions

TL;DR

Shap-E is a novel generative model that creates diverse 3D assets by directly producing implicit function parameters, enabling fast, high-quality 3D object generation from text or other inputs.

Contribution

It introduces a two-stage training process for a conditional diffusion model that generates 3D assets as implicit functions, improving speed and quality over previous point cloud methods.

Findings

Generates complex, diverse 3D assets in seconds.

Converges faster and achieves comparable or better quality than Point-E.

Models a higher-dimensional, multi-representation output space.

Abstract

We present Shap-E, a conditional generative model for 3D assets. Unlike recent work on 3D generative models which produce a single output representation, Shap-E directly generates the parameters of implicit functions that can be rendered as both textured meshes and neural radiance fields. We train Shap-E in two stages: first, we train an encoder that deterministically maps 3D assets into the parameters of an implicit function; second, we train a conditional diffusion model on outputs of the encoder. When trained on a large dataset of paired 3D and text data, our resulting models are capable of generating complex and diverse 3D assets in a matter of seconds. When compared to Point-E, an explicit generative model over point clouds, Shap-E converges faster and reaches comparable or better sample quality despite modeling a higher-dimensional, multi-representation output space. We release model weights, inference code, and samples at https://github.com/openai/shap-e.