Interactive Segmentation of Radiance Fields

TL;DR

This paper introduces ISRF, an interactive method for segmenting objects in radiance fields with high accuracy, enabling advanced scene manipulation and compositing in mixed reality applications.

Contribution

The paper presents a novel interactive segmentation approach for radiance fields that handles complex objects with fine details, outperforming prior methods in accuracy and usability.

Findings

Achieves state-of-the-art segmentation accuracy in radiance fields.

Enables effective object compositing and appearance modification.

Provides an accessible interactive segmentation tool for practical use.

Abstract

Radiance Fields (RF) are popular to represent casually-captured scenes for new view synthesis and several applications beyond it. Mixed reality on personal spaces needs understanding and manipulating scenes represented as RFs, with semantic segmentation of objects as an important step. Prior segmentation efforts show promise but don't scale to complex objects with diverse appearance. We present the ISRF method to interactively segment objects with fine structure and appearance. Nearest neighbor feature matching using distilled semantic features identifies high-confidence seed regions. Bilateral search in a joint spatio-semantic space grows the region to recover accurate segmentation. We show state-of-the-art results of segmenting objects from RFs and compositing them to another scene, changing appearance, etc., and an interactive segmentation tool that others can use. Project Page: https://rahul-goel.github.io/isrf/