HexaGen3D: StableDiffusion is just one step away from Fast and Diverse Text-to-3D Generation

TL;DR

HexaGen3D leverages pretrained 2D diffusion models to efficiently generate high-quality, diverse 3D assets from text prompts in just 7 seconds, overcoming data scarcity issues.

Contribution

It introduces a novel method that fine-tunes a 2D diffusion model to predict 3D projections and latents, enabling fast, high-quality 3D generation without per-sample optimization.

Findings

Generates 3D assets in 7 seconds with high quality.

Outperforms existing methods in quality-to-latency ratio.

Shows strong generalization to new objects and compositions.

Abstract

Despite the latest remarkable advances in generative modeling, efficient generation of high-quality 3D assets from textual prompts remains a difficult task. A key challenge lies in data scarcity: the most extensive 3D datasets encompass merely millions of assets, while their 2D counterparts contain billions of text-image pairs. To address this, we propose a novel approach which harnesses the power of large, pretrained 2D diffusion models. More specifically, our approach, HexaGen3D, fine-tunes a pretrained text-to-image model to jointly predict 6 orthographic projections and the corresponding latent triplane. We then decode these latents to generate a textured mesh. HexaGen3D does not require per-sample optimization, and can infer high-quality and diverse objects from textual prompts in 7 seconds, offering significantly better quality-to-latency trade-offs when comparing to existing approaches. Furthermore, HexaGen3D demonstrates strong generalization to new objects or compositions.