Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation

TL;DR

Prometheus is a novel 3D-aware latent diffusion model that enables fast, high-fidelity text-to-3D scene generation by leveraging multi-view datasets and an RGB-D latent space for disentangling appearance and geometry.

Contribution

It introduces a 3D-aware latent diffusion framework for text-to-3D generation, utilizing minimal adjustments to pre-trained models and an RGB-D latent space for improved fidelity.

Findings

Effective in feed-forward 3D Gaussian reconstruction

Achieves high-quality text-to-3D scene generation

Demonstrates generalizability across datasets

Abstract

In this work, we introduce Prometheus, a 3D-aware latent diffusion model for text-to-3D generation at both object and scene levels in seconds. We formulate 3D scene generation as multi-view, feed-forward, pixel-aligned 3D Gaussian generation within the latent diffusion paradigm. To ensure generalizability, we build our model upon pre-trained text-to-image generation model with only minimal adjustments, and further train it using a large number of images from both single-view and multi-view datasets. Furthermore, we introduce an RGB-D latent space into 3D Gaussian generation to disentangle appearance and geometry information, enabling efficient feed-forward generation of 3D Gaussians with better fidelity and geometry. Extensive experimental results demonstrate the effectiveness of our method in both feed-forward 3D Gaussian reconstruction and text-to-3D generation. Project page: https://freemty.github.io/project-prometheus/