NeRF-RPN: A general framework for object detection in NeRFs

TL;DR

NeRF-RPN introduces a novel framework that detects 3D object bounding boxes directly within NeRF representations using multi-scale volumetric features, eliminating the need for rendering at specific viewpoints.

Contribution

It is the first framework to perform object detection directly on NeRFs, utilizing a new voxel-based feature representation and enabling end-to-end training without class labels.

Findings

Effective 3D bounding box regression without rendering

Versatile framework applicable to various architectures

Established a new benchmark dataset for NeRF object detection

Abstract

This paper presents the first significant object detection framework, NeRF-RPN, which directly operates on NeRF. Given a pre-trained NeRF model, NeRF-RPN aims to detect all bounding boxes of objects in a scene. By exploiting a novel voxel representation that incorporates multi-scale 3D neural volumetric features, we demonstrate it is possible to regress the 3D bounding boxes of objects in NeRF directly without rendering the NeRF at any viewpoint. NeRF-RPN is a general framework and can be applied to detect objects without class labels. We experimented NeRF-RPN with various backbone architectures, RPN head designs and loss functions. All of them can be trained in an end-to-end manner to estimate high quality 3D bounding boxes. To facilitate future research in object detection for NeRF, we built a new benchmark dataset which consists of both synthetic and real-world data with careful labeling and clean up. Code and dataset are available at https://github.com/lyclyc52/NeRF_RPN.