DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editing

TL;DR

DGE introduces a two-stage method for efficient, multi-view consistent 3D editing guided by language, improving accuracy and speed over traditional iterative approaches by leveraging 3D Gaussian Splatting.

Contribution

The paper presents a training-free approach to make 2D image editors multi-view consistent and directly optimizes 3D representations, enabling fast, accurate 3D scene editing from language instructions.

Findings

Significantly faster than existing methods

Achieves multi-view consistency without training

Allows selective editing of scene parts

Abstract

We consider the problem of editing 3D objects and scenes based on open-ended language instructions. A common approach to this problem is to use a 2D image generator or editor to guide the 3D editing process, obviating the need for 3D data. However, this process is often inefficient due to the need for iterative updates of costly 3D representations, such as neural radiance fields, either through individual view edits or score distillation sampling. A major disadvantage of this approach is the slow convergence caused by aggregating inconsistent information across views, as the guidance from 2D models is not multi-view consistent. We thus introduce the Direct Gaussian Editor (DGE), a method that addresses these issues in two stages. First, we modify a given high-quality image editor like InstructPix2Pix to be multi-view consistent. To do so, we propose a training-free approach that integrates cues from the 3D geometry of the underlying scene. Second, given a multi-view consistent edited sequence of images, we directly and efficiently optimize the 3D representation, which is based on 3D Gaussian Splatting. Because it avoids incremental and iterative edits, DGE is significantly more accurate and efficient than existing approaches and offers additional benefits, such as enabling selective editing of parts of the scene.