Mastering Regional 3DGS: Locating, Initializing, and Editing with Diverse 2D Priors

TL;DR

This paper introduces a novel method for precise regional editing of 3D scenes represented by Gaussian splatting, leveraging 2D diffusion models and inverse rendering to improve accuracy, consistency, and speed in local 3D scene modifications.

Contribution

It proposes a new approach combining 2D diffusion editing and inverse rendering for accurate, view-consistent 3D scene editing with enhanced efficiency and fidelity.

Findings

Achieves state-of-the-art performance in 3D local editing.

Provides up to 4x speedup over existing methods.

Ensures coherent multi-view editing through iterative refinement.

Abstract

Many 3D scene editing tasks focus on modifying local regions rather than the entire scene, except for some global applications like style transfer, and in the context of 3D Gaussian Splatting (3DGS), where scenes are represented by a series of Gaussians, this structure allows for precise regional edits, offering enhanced control over specific areas of the scene; however, the challenge lies in the fact that 3D semantic parsing often underperforms compared to its 2D counterpart, making targeted manipulations within 3D spaces more difficult and limiting the fidelity of edits, which we address by leveraging 2D diffusion editing to accurately identify modification regions in each view, followed by inverse rendering for 3D localization, then refining the frontal view and initializing a coarse 3DGS with consistent views and approximate shapes derived from depth maps predicted by a 2D foundation model, thereby supporting an iterative, view-consistent editing process that gradually enhances structural details and textures to ensure coherence across perspectives. Experiments demonstrate that our method achieves state-of-the-art performance while delivering up to a $4\times$ speedup, providing a more efficient and effective approach to 3D scene local editing.