Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation

TL;DR

Zero-1-to-G leverages pretrained 2D diffusion models to enable direct 3D object generation from single views using Gaussian splats, incorporating cross-view attention for 3D consistency.

Contribution

It introduces the first method to utilize pretrained 2D diffusion priors for direct 3D generation via Gaussian splats with cross-view attention mechanisms.

Findings

Achieves superior 3D generation quality on synthetic and real datasets.

Effectively captures 3D consistency through novel attention layers.

Demonstrates efficient training and strong generalization to unseen objects.

Abstract

Recent advances in 2D image generation have achieved remarkable quality,largely driven by the capacity of diffusion models and the availability of large-scale datasets. However, direct 3D generation is still constrained by the scarcity and lower fidelity of 3D datasets. In this paper, we introduce Zero-1-to-G, a novel approach that addresses this problem by enabling direct single-view generation on Gaussian splats using pretrained 2D diffusion models. Our key insight is that Gaussian splats, a 3D representation, can be decomposed into multi-view images encoding different attributes. This reframes the challenging task of direct 3D generation within a 2D diffusion framework, allowing us to leverage the rich priors of pretrained 2D diffusion models. To incorporate 3D awareness, we introduce cross-view and cross-attribute attention layers, which capture complex correlations and enforce 3D consistency across generated splats. This makes Zero-1-to-G the first direct image-to-3D generative model to effectively utilize pretrained 2D diffusion priors, enabling efficient training and improved generalization to unseen objects. Extensive experiments on both synthetic and in-the-wild datasets demonstrate superior performance in 3D object generation, offering a new approach to high-quality 3D generation.