ITS3D: Inference-Time Scaling for Text-Guided 3D Diffusion Models

TL;DR

ITS3D introduces an inference-time optimization framework for text-guided 3D diffusion models, significantly improving generation quality without additional training by refining Gaussian noise inputs through a verifier-guided search.

Contribution

The paper presents a novel inference-time scaling method, ITS3D, with techniques for stability, efficiency, and exploration in 3D generative modeling, advancing the quality of text-to-3D synthesis.

Findings

Enhanced 3D generation quality demonstrated

Verifier-guided search effectively refines noise inputs

Techniques improve stability and exploration in high-dimensional space

Abstract

We explore inference-time scaling in text-guided 3D diffusion models to enhance generative quality without additional training. To this end, we introduce ITS3D, a framework that formulates the task as an optimization problem to identify the most effective Gaussian noise input. The framework is driven by a verifier-guided search algorithm, where the search algorithm iteratively refines noise candidates based on verifier feedback. To address the inherent challenges of 3D generation, we introduce three techniques for improved stability, efficiency, and exploration capability. 1) Gaussian normalization is applied to stabilize the search process. It corrects distribution shifts when noise candidates deviate from a standard Gaussian distribution during iterative updates. 2) The high-dimensional nature of the 3D search space increases computational complexity. To mitigate this, a singular value decomposition-based compression technique is employed to reduce dimensionality while preserving effective search directions. 3) To further prevent convergence to suboptimal local minima, a singular space reset mechanism dynamically updates the search space based on diversity measures. Extensive experiments demonstrate that ITS3D enhances text-to-3D generation quality, which shows the potential of computationally efficient search methods in generative processes. The source code is available at https://github.com/ZhenglinZhou/ITS3D.