ArtisanGS: Interactive Tools for Gaussian Splat Selection with AI and Human in the Loop

TL;DR

This paper introduces an interactive toolset for Gaussian Splat selection and segmentation, combining AI-driven propagation and manual editing to improve 3D scene manipulation from real-world captures.

Contribution

It presents a novel interactive suite with AI-assisted and manual tools for versatile Gaussian Splat selection and editing, addressing limitations of automatic methods.

Findings

Outperforms state-of-the-art Gaussian Splat selection methods.

Enables precise user-guided local editing of 3D scenes.

Applicable to in-the-wild captures without extra optimization.

Abstract

Representation in the family of 3D Gaussian Splats (3DGS) are growing into a viable alternative to traditional graphics for an expanding number of application, including recent techniques that facilitate physics simulation and animation. However, extracting usable objects from in-the-wild captures remains challenging and controllable editing techniques for this representation are limited. Unlike the bulk of emerging techniques, focused on automatic solutions or high-level editing, we introduce an interactive suite of tools centered around versatile Gaussian Splat selection and segmentation. We propose a fast AI-driven method to propagate user-guided 2D selection masks to 3DGS selections. This technique allows for user intervention in the case of errors and is further coupled with flexible manual selection and segmentation tools. These allow a user to achieve virtually any binary segmentation of an unstructured 3DGS scene. We evaluate our toolset against the state-of-the-art for Gaussian Splat selection and demonstrate their utility for downstream applications by developing a user-guided local editing approach, leveraging a custom Video Diffusion Model. With flexible selection tools, users have direct control over the areas that the AI can modify. Our selection and editing tools can be used for any in-the-wild capture without additional optimization.