GSEditPro: 3D Gaussian Splatting Editing with Attention-based Progressive Localization

TL;DR

GSEditPro introduces an attention-based progressive localization framework for precise, text-driven editing of 3D scenes using Gaussian Splatting, overcoming localization and manipulation limitations of previous methods.

Contribution

The paper presents a novel 3D scene editing framework that employs an attention-based localization module and an optimization method for accurate, flexible, text-guided editing of 3D Gaussian Splatting scenes.

Findings

Effective localization of editing areas via semantic labels on Gaussians

Stable and refined editing results guided by Score Distillation Sampling

Demonstrated superior performance through extensive experiments

Abstract

With the emergence of large-scale Text-to-Image(T2I) models and implicit 3D representations like Neural Radiance Fields (NeRF), many text-driven generative editing methods based on NeRF have appeared. However, the implicit encoding of geometric and textural information poses challenges in accurately locating and controlling objects during editing. Recently, significant advancements have been made in the editing methods of 3D Gaussian Splatting, a real-time rendering technology that relies on explicit representation. However, these methods still suffer from issues including inaccurate localization and limited manipulation over editing. To tackle these challenges, we propose GSEditPro, a novel 3D scene editing framework which allows users to perform various creative and precise editing using text prompts only. Leveraging the explicit nature of the 3D Gaussian distribution, we introduce an attention-based progressive localization module to add semantic labels to each Gaussian during rendering. This enables precise localization on editing areas by classifying Gaussians based on their relevance to the editing prompts derived from cross-attention layers of the T2I model. Furthermore, we present an innovative editing optimization method based on 3D Gaussian Splatting, obtaining stable and refined editing results through the guidance of Score Distillation Sampling and pseudo ground truth. We prove the efficacy of our method through extensive experiments.