Instance Neural Radiance Field

TL;DR

This paper introduces Instance NeRF, a learning-based 3D instance segmentation method that extends NeRF by generating consistent 3D masks and querying instance info at any point, improving segmentation and manipulation.

Contribution

It presents a novel 3D instance segmentation pipeline for NeRF that combines volumetric features, mask prediction, and multi-view supervision, enabling pure inference segmentation.

Findings

Outperforms previous NeRF segmentation methods on synthetic and real datasets.

Achieves consistent 2D segmentation across multiple views.

Enables querying instance information at any 3D point.

Abstract

This paper presents one of the first learning-based NeRF 3D instance segmentation pipelines, dubbed as Instance Neural Radiance Field, or Instance NeRF. Taking a NeRF pretrained from multi-view RGB images as input, Instance NeRF can learn 3D instance segmentation of a given scene, represented as an instance field component of the NeRF model. To this end, we adopt a 3D proposal-based mask prediction network on the sampled volumetric features from NeRF, which generates discrete 3D instance masks. The coarse 3D mask prediction is then projected to image space to match 2D segmentation masks from different views generated by existing panoptic segmentation models, which are used to supervise the training of the instance field. Notably, beyond generating consistent 2D segmentation maps from novel views, Instance NeRF can query instance information at any 3D point, which greatly enhances NeRF object segmentation and manipulation. Our method is also one of the first to achieve such results in pure inference. Experimented on synthetic and real-world NeRF datasets with complex indoor scenes, Instance NeRF surpasses previous NeRF segmentation works and competitive 2D segmentation methods in segmentation performance on unseen views. Watch the demo video at https://youtu.be/wW9Bme73coI. Code and data are available at https://github.com/lyclyc52/Instance_NeRF.