Dreamer XL: Towards High-Resolution Text-to-3D Generation via Trajectory Score Matching

TL;DR

This paper introduces Trajectory Score Matching (TSM), a novel method that improves the stability and quality of high-resolution text-to-3D generation by reducing error accumulation in diffusion models, and enhances multi-stage optimization with Stable Diffusion XL.

Contribution

The paper proposes TSM to address pseudo ground truth inconsistency in diffusion-based 3D generation and integrates Stable Diffusion XL with pixel-wise gradient clipping for better high-resolution results.

Findings

TSM reduces error accumulation compared to ISM.

The method surpasses state-of-the-art in visual quality.

Stable Diffusion XL with gradient clipping improves 3D generation stability.

Abstract

In this work, we propose a novel Trajectory Score Matching (TSM) method that aims to solve the pseudo ground truth inconsistency problem caused by the accumulated error in Interval Score Matching (ISM) when using the Denoising Diffusion Implicit Models (DDIM) inversion process. Unlike ISM which adopts the inversion process of DDIM to calculate on a single path, our TSM method leverages the inversion process of DDIM to generate two paths from the same starting point for calculation. Since both paths start from the same starting point, TSM can reduce the accumulated error compared to ISM, thus alleviating the problem of pseudo ground truth inconsistency. TSM enhances the stability and consistency of the model's generated paths during the distillation process. We demonstrate this experimentally and further show that ISM is a special case of TSM. Furthermore, to optimize the current multi-stage optimization process from high-resolution text to 3D generation, we adopt Stable Diffusion XL for guidance. In response to the issues of abnormal replication and splitting caused by unstable gradients during the 3D Gaussian splatting process when using Stable Diffusion XL, we propose a pixel-by-pixel gradient clipping method. Extensive experiments show that our model significantly surpasses the state-of-the-art models in terms of visual quality and performance. Code: \url{https://github.com/xingy038/Dreamer-XL}.