Seg-Wild: Interactive Segmentation based on 3D Gaussian Splatting for Unconstrained Image Collections

TL;DR

Seg-Wild introduces an interactive 3D Gaussian Splatting-based segmentation method tailored for unconstrained, in-the-wild image collections, effectively handling lighting inconsistencies and occlusions to improve scene reconstruction and segmentation quality.

Contribution

The paper presents Seg-Wild, a novel interactive segmentation approach using 3D Gaussian Splatting and a new smoothing technique, addressing challenges in unconstrained scene segmentation.

Findings

Outperforms previous methods in segmentation accuracy

Enhances scene reconstruction quality

Introduces a new benchmark for in-the-wild scene segmentation

Abstract

Reconstructing and segmenting scenes from unconstrained photo collections obtained from the Internet is a novel but challenging task. Unconstrained photo collections are easier to get than well-captured photo collections. These unconstrained images suffer from inconsistent lighting and transient occlusions, which makes segmentation challenging. Previous segmentation methods cannot address transient occlusions or accurately restore the scene's lighting conditions. Therefore, we propose Seg-Wild, an interactive segmentation method based on 3D Gaussian Splatting for unconstrained image collections, suitable for in-the-wild scenes. We integrate multi-dimensional feature embeddings for each 3D Gaussian and calculate the feature similarity between the feature embeddings and the segmentation target to achieve interactive segmentation in the 3D scene. Additionally, we introduce the Spiky 3D Gaussian Cutter (SGC) to smooth abnormal 3D Gaussians. We project the 3D Gaussians onto a 2D plane and calculate the ratio of 3D Gaussians that need to be cut using the SAM mask. We also designed a benchmark to evaluate segmentation quality in in-the-wild scenes. Experimental results demonstrate that compared to previous methods, Seg-Wild achieves better segmentation results and reconstruction quality. Our code will be available at https://github.com/Sugar0725/Seg-Wild.