3DSceneEditor: Controllable 3D Scene Editing with Gaussian Splatting

TL;DR

3DSceneEditor introduces a novel 3D pipeline using Gaussian Splatting for precise, interactive editing of complex 3D scenes, significantly improving control and efficiency over existing multi-step, diffusion-based methods.

Contribution

It presents a fully 3D-based editing framework that enables direct Gaussian manipulation for high-quality scene edits, integrating semantic labeling and zero-shot grounding for improved control.

Findings

Outperforms existing methods in editing precision

Achieves higher efficiency and interactive performance

Establishes new benchmarks for 3D scene editing

Abstract

The creation of 3D scenes has traditionally been both labor-intensive and costly, requiring designers to meticulously configure 3D assets and environments. Recent advancements in generative AI, including text-to-3D and image-to-3D methods, have dramatically reduced the complexity and cost of this process. However, current techniques for editing complex 3D scenes continue to rely on generally interactive multi-step, 2D-to-3D projection methods and diffusion-based techniques, which often lack precision in control and hamper interactive-rate performance. In this work, we propose ***3DSceneEditor***, a fully 3D-based paradigm for interactive-rate, precise editing of intricate 3D scenes using Gaussian Splatting. Unlike conventional methods, 3DSceneEditor operates through a streamlined 3D pipeline, enabling direct Gaussian-based manipulation for efficient, high-quality edits based on input prompts. The proposed framework (i) integrates a pre-trained instance segmentation model for semantic labeling; (ii) employs a zero-shot grounding approach with CLIP to align target objects with user prompts; and (iii) applies scene modifications, such as object addition, repositioning, recoloring, replacing, and removal--directly on Gaussians. Extensive experimental results show that 3DSceneEditor surpasses existing state-of-the-art techniques in terms of both editing precision and efficiency, establishing a new benchmark for efficient and interactive 3D scene customization.