Open-NeRF: Towards Open Vocabulary NeRF Decomposition

TL;DR

Open-NeRF introduces a method that combines large-scale segmentation models with hierarchical embeddings to enable accurate 3D object decomposition from open vocabulary queries, improving flexibility and precision in NeRF applications.

Contribution

The paper presents Open-NeRF, a novel approach that integrates foundation models with hierarchical embeddings for open-vocabulary NeRF decomposition, enhancing both flexibility and accuracy.

Findings

Outperforms state-of-the-art methods like LERF and FFD in open-vocabulary scenarios.

Achieves consistent object recognition across viewpoints, even with occlusions.

Enables applications in robotics and vision-language interaction in open-world 3D scenes.

Abstract

In this paper, we address the challenge of decomposing Neural Radiance Fields (NeRF) into objects from an open vocabulary, a critical task for object manipulation in 3D reconstruction and view synthesis. Current techniques for NeRF decomposition involve a trade-off between the flexibility of processing open-vocabulary queries and the accuracy of 3D segmentation. We present, Open-vocabulary Embedded Neural Radiance Fields (Open-NeRF), that leverage large-scale, off-the-shelf, segmentation models like the Segment Anything Model (SAM) and introduce an integrate-and-distill paradigm with hierarchical embeddings to achieve both the flexibility of open-vocabulary querying and 3D segmentation accuracy. Open-NeRF first utilizes large-scale foundation models to generate hierarchical 2D mask proposals from varying viewpoints. These proposals are then aligned via tracking approaches and integrated within the 3D space and subsequently distilled into the 3D field. This process ensures consistent recognition and granularity of objects from different viewpoints, even in challenging scenarios involving occlusion and indistinct features. Our experimental results show that the proposed Open-NeRF outperforms state-of-the-art methods such as LERF \cite{lerf} and FFD \cite{ffd} in open-vocabulary scenarios. Open-NeRF offers a promising solution to NeRF decomposition, guided by open-vocabulary queries, enabling novel applications in robotics and vision-language interaction in open-world 3D scenes.