Geometry-Aware Score Distillation via 3D Consistent Noising and Gradient Consistency Modeling

TL;DR

This paper introduces GSD, a framework that enhances 3D consistency in score distillation sampling for text-to-3D generation by addressing geometric inconsistencies through noising, gradient warping, and a new loss.

Contribution

GSD provides a simple, general, and plug-and-play solution to incorporate 3D consistency into SDS, effectively solving geometric inconsistency issues in text-to-3D generation.

Findings

Significantly improves 3D consistency in text-to-3D models

Addresses Janus problem with minimal additional computation

Compatible with existing score distillation models

Abstract

Score distillation sampling (SDS), the methodology in which the score from pretrained 2D diffusion models is distilled into 3D representation, has recently brought significant advancements in text-to-3D generation task. However, this approach is still confronted with critical geometric inconsistency problems such as the Janus problem. Starting from a hypothesis that such inconsistency problems may be induced by multiview inconsistencies between 2D scores predicted from various viewpoints, we introduce GSD, a simple and general plug-and-play framework for incorporating 3D consistency and therefore geometry awareness into the SDS process. Our methodology is composed of three components: 3D consistent noising, designed to produce 3D consistent noise maps that perfectly follow the standard Gaussian distribution, geometry-based gradient warping for identifying correspondences between predicted gradients of different viewpoints, and novel gradient consistency loss to optimize the scene geometry toward producing more consistent gradients. We demonstrate that our method significantly improves performance, successfully addressing the geometric inconsistency problems in text-to-3D generation task with minimal computation cost and being compatible with existing score distillation-based models. Our project page is available at https://ku-cvlab.github.io/GSD/.