Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation

TL;DR

This paper introduces Radiance Field Propagation (RFP), an unsupervised method for segmenting objects in 3D scenes reconstructed from multi-view images using neural radiance fields, without requiring annotations or prior knowledge.

Contribution

The paper proposes a novel unsupervised segmentation approach for 3D scenes using neural radiance fields, including a propagation strategy and iterative refinement, advancing 3D scene understanding.

Findings

RFP achieves more accurate segmentation than previous unsupervised methods.

Segmentation results are comparable to supervised NeRF-based methods.

Object representations enable 3D object editing.

Abstract

We present radiance field propagation (RFP), a novel approach to segmenting objects in 3D during reconstruction given only unlabeled multi-view images of a scene. RFP is derived from emerging neural radiance field-based techniques, which jointly encodes semantics with appearance and geometry. The core of our method is a novel propagation strategy for individual objects' radiance fields with a bidirectional photometric loss, enabling an unsupervised partitioning of a scene into salient or meaningful regions corresponding to different object instances. To better handle complex scenes with multiple objects and occlusions, we further propose an iterative expectation-maximization algorithm to refine object masks. RFP is one of the first unsupervised approach for tackling 3D real scene object segmentation for neural radiance field (NeRF) without any supervision, annotations, or other cues such as 3D bounding boxes and prior knowledge of object class. Experiments demonstrate that RFP achieves feasible segmentation results that are more accurate than previous unsupervised image/scene segmentation approaches, and are comparable to existing supervised NeRF-based methods. The segmented object representations enable individual 3D object editing operations.