Neural Volumetric Object Selection

TL;DR

This paper presents a novel method for 3D object segmentation in neural volumetric representations using user scribbles, leveraging a new voxel embedding technique and a dedicated dataset, outperforming existing methods.

Contribution

It introduces a new approach for 3D object selection in neural volumetric models using multi-view user input and a novel voxel embedding, along with a new dataset for evaluation.

Findings

Outperforms baseline segmentation methods

Effective in rendering objects from novel views

Demonstrates robustness with real-world multi-view data

Abstract

We introduce an approach for selecting objects in neural volumetric 3D representations, such as multi-plane images (MPI) and neural radiance fields (NeRF). Our approach takes a set of foreground and background 2D user scribbles in one view and automatically estimates a 3D segmentation of the desired object, which can be rendered into novel views. To achieve this result, we propose a novel voxel feature embedding that incorporates the neural volumetric 3D representation and multi-view image features from all input views. To evaluate our approach, we introduce a new dataset of human-provided segmentation masks for depicted objects in real-world multi-view scene captures. We show that our approach out-performs strong baselines, including 2D segmentation and 3D segmentation approaches adapted to our task.