TV-3DG: Mastering Text-to-3D Customized Generation with Visual Prompt

TL;DR

This paper introduces TV-3DG, a novel method for text-to-3D generation that effectively incorporates visual prompts and improves quality by addressing limitations of existing score distillation sampling techniques.

Contribution

We propose Classifier Score Matching (CSM) to replace SDS, enabling better multi-condition customization, and integrate visual prompts with attention and calibration modules for high-quality 3D generation.

Findings

CSM reduces noise and improves stability in 3D generation.

Visual prompts enhance customization and detail in generated 3D models.

Experiments show TV-3DG outperforms previous methods in quality and stability.

Abstract

In recent years, advancements in generative models have significantly expanded the capabilities of text-to-3D generation. Many approaches rely on Score Distillation Sampling (SDS) technology. However, SDS struggles to accommodate multi-condition inputs, such as text and visual prompts, in customized generation tasks. To explore the core reasons, we decompose SDS into a difference term and a classifier-free guidance term. Our analysis identifies the core issue as arising from the difference term and the random noise addition during the optimization process, both contributing to deviations from the target mode during distillation. To address this, we propose a novel algorithm, Classifier Score Matching (CSM), which removes the difference term in SDS and uses a deterministic noise addition process to reduce noise during optimization, effectively overcoming the low-quality limitations of SDS in our customized generation framework. Based on CSM, we integrate visual prompt information with an attention fusion mechanism and sampling guidance techniques, forming the Visual Prompt CSM (VPCSM) algorithm. Furthermore, we introduce a Semantic-Geometry Calibration (SGC) module to enhance quality through improved textual information integration. We present our approach as TV-3DG, with extensive experiments demonstrating its capability to achieve stable, high-quality, customized 3D generation. Project page: \url{https://yjhboy.github.io/TV-3DG}