Scene-Generalizable Interactive Segmentation of Radiance Fields

TL;DR

This paper introduces a novel scene-generalizable interactive segmentation method for radiance fields that effectively segments 3D objects in unseen scenes using minimal user input, overcoming scene-specific limitations.

Contribution

The paper proposes SGISRF, a new approach enabling 3D object segmentation in unseen radiance field scenes guided by few user clicks, with three innovative techniques addressing key challenges.

Findings

Effective scene-generalization demonstrated on real-world benchmarks.

Outperforms classical scene-specific optimization methods.

Achieves accurate 3D segmentation with minimal user interaction.

Abstract

Existing methods for interactive segmentation in radiance fields entail scene-specific optimization and thus cannot generalize across different scenes, which greatly limits their applicability. In this work we make the first attempt at Scene-Generalizable Interactive Segmentation in Radiance Fields (SGISRF) and propose a novel SGISRF method, which can perform 3D object segmentation for novel (unseen) scenes represented by radiance fields, guided by only a few interactive user clicks in a given set of multi-view 2D images. In particular, the proposed SGISRF focuses on addressing three crucial challenges with three specially designed techniques. First, we devise the Cross-Dimension Guidance Propagation to encode the scarce 2D user clicks into informative 3D guidance representations. Second, the Uncertainty-Eliminated 3D Segmentation module is designed to achieve efficient yet effective 3D segmentation. Third, Concealment-Revealed Supervised Learning scheme is proposed to reveal and correct the concealed 3D segmentation errors resulted from the supervision in 2D space with only 2D mask annotations. Extensive experiments on two real-world challenging benchmarks covering diverse scenes demonstrate 1) effectiveness and scene-generalizability of the proposed method, 2) favorable performance compared to classical method requiring scene-specific optimization.