Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D Prior

TL;DR

Sherpa3D leverages coarse 3D priors to enhance text-to-3D generation, improving quality, diversity, and geometric consistency by guiding 2D diffusion models with structural and semantic cues.

Contribution

It introduces a novel framework that exploits coarse 3D knowledge to guide 2D diffusion-based text-to-3D generation, addressing multi-face issues and enhancing output fidelity.

Findings

Outperforms state-of-the-art methods in quality and 3D consistency.

Enables high-fidelity and diverse 3D asset generation from text prompts.

Effectively mitigates view-agnostic ambiguity in 3D generation.

Abstract

Recently, 3D content creation from text prompts has demonstrated remarkable progress by utilizing 2D and 3D diffusion models. While 3D diffusion models ensure great multi-view consistency, their ability to generate high-quality and diverse 3D assets is hindered by the limited 3D data. In contrast, 2D diffusion models find a distillation approach that achieves excellent generalization and rich details without any 3D data. However, 2D lifting methods suffer from inherent view-agnostic ambiguity thereby leading to serious multi-face Janus issues, where text prompts fail to provide sufficient guidance to learn coherent 3D results. Instead of retraining a costly viewpoint-aware model, we study how to fully exploit easily accessible coarse 3D knowledge to enhance the prompts and guide 2D lifting optimization for refinement. In this paper, we propose Sherpa3D, a new text-to-3D framework that achieves high-fidelity, generalizability, and geometric consistency simultaneously. Specifically, we design a pair of guiding strategies derived from the coarse 3D prior generated by the 3D diffusion model: a structural guidance for geometric fidelity and a semantic guidance for 3D coherence. Employing the two types of guidance, the 2D diffusion model enriches the 3D content with diversified and high-quality results. Extensive experiments show the superiority of our Sherpa3D over the state-of-the-art text-to-3D methods in terms of quality and 3D consistency.