Gradient-Driven 3D Segmentation and Affordance Transfer in Gaussian Splatting Using 2D Masks

TL;DR

This paper presents a novel voting-based method that extends 2D segmentation to 3D Gaussian splats, enabling accurate segmentation, Gaussian pruning, and few-shot affordance transfer for applications like AR and robotics.

Contribution

It introduces a new voting-based approach leveraging masked gradients for 3D segmentation and Gaussian pruning, along with a method for few-shot affordance transfer from 2D to 3D.

Findings

Achieved up to 21% Gaussian compression through gradient-based pruning.

Enabled effective transfer of 2D annotations to 3D Gaussian splats.

Demonstrated applicability in AR, object editing, and robotics.

Abstract

3D Gaussian Splatting has emerged as a powerful 3D scene representation technique, capturing fine details with high efficiency. In this paper, we introduce a novel voting-based method that extends 2D segmentation models to 3D Gaussian splats. Our approach leverages masked gradients, where gradients are filtered by input 2D masks, and these gradients are used as votes to achieve accurate segmentation. As a byproduct, we discovered that inference-time gradients can also be used to prune Gaussians, resulting in up to 21% compression. Additionally, we explore few-shot affordance transfer, allowing annotations from 2D images to be effectively transferred onto 3D Gaussian splats. The robust yet straightforward mathematical formulation underlying this approach makes it a highly effective tool for numerous downstream applications, such as augmented reality (AR), object editing, and robotics. The project code and additional resources are available at https://jojijoseph.github.io/3dgs-segmentation.