AG$^2$aussian: Anchor-Graph Structured Gaussian Splatting for Instance-Level 3D Scene Understanding and Editing

TL;DR

AG$^2$aussian introduces an anchor-graph structure to organize and regulate Gaussian primitives in 3D scene understanding, improving semantic accuracy and enabling effective editing and querying tasks.

Contribution

It proposes a novel anchor-graph framework for semantic-aware Gaussian representations, enhancing instance-level organization and propagation in 3D Gaussian Splatting.

Findings

Improves instance-aware Gaussian selection accuracy.

Enhances performance in interactive and text-driven scene queries.

Facilitates object removal and physics-based editing.

Abstract

3D Gaussian Splatting (3DGS) has witnessed exponential adoption across diverse applications, driving a critical need for semantic-aware 3D Gaussian representations to enable scene understanding and editing tasks. Existing approaches typically attach semantic features to a collection of free Gaussians and distill the features via differentiable rendering, leading to noisy segmentation and a messy selection of Gaussians. In this paper, we introduce AG$^2$aussian, a novel framework that leverages an anchor-graph structure to organize semantic features and regulate Gaussian primitives. Our anchor-graph structure not only promotes compact and instance-aware Gaussian distributions, but also facilitates graph-based propagation, achieving a clean and accurate instance-level Gaussian selection. Extensive validation across four applications, i.e. interactive click-based query, open-vocabulary text-driven query, object removal editing, and physics simulation, demonstrates the advantages of our approach and its benefits to various applications. The experiments and ablation studies further evaluate the effectiveness of the key designs of our approach.