Rethinking Score Distilling Sampling for 3D Editing and Generation

TL;DR

This paper introduces Unified Distillation Sampling (UDS), a novel method that unifies 3D asset generation and editing by refining gradient terms in Score Distillation Sampling, leading to improved performance in both tasks.

Contribution

The paper proposes UDS, a new approach that combines generation and editing capabilities in 3D asset synthesis by unifying gradient processes in SDS variants.

Findings

UDS outperforms baseline methods in 3D generation quality.

UDS demonstrates superior editing capabilities for 3D assets.

The method effectively bridges the gap between 3D generation and editing.

Abstract

Score Distillation Sampling (SDS) has emerged as a prominent method for text-to-3D generation by leveraging the strengths of 2D diffusion models. However, SDS is limited to generation tasks and lacks the capability to edit existing 3D assets. Conversely, variants of SDS that introduce editing capabilities often can not generate new 3D assets effectively. In this work, we observe that the processes of generation and editing within SDS and its variants have unified underlying gradient terms. Building on this insight, we propose Unified Distillation Sampling (UDS), a method that seamlessly integrates both the generation and editing of 3D assets. Essentially, UDS refines the gradient terms used in vanilla SDS methods, unifying them to support both tasks. Extensive experiments demonstrate that UDS not only outperforms baseline methods in generating 3D assets with richer details but also excels in editing tasks, thereby bridging the gap between 3D generation and editing. The code is available on: https://github.com/xingy038/UDS.