PeRFception: Perception using Radiance Fields

TL;DR

This paper introduces PeRFception, a large-scale dataset and models for perception tasks using implicit 3D representations from Neural Radiance Fields, enabling efficient 3D understanding and synthesis.

Contribution

It creates the first large-scale implicit 3D perception dataset using NeRF variants and develops models that directly operate on this compact, unified 2D-3D data format.

Findings

Achieved 96.4% memory compression from original datasets.

Developed perception models that process implicit 3D data directly.

Proposed augmentation techniques to improve model robustness.

Abstract

The recent progress in implicit 3D representation, i.e., Neural Radiance Fields (NeRFs), has made accurate and photorealistic 3D reconstruction possible in a differentiable manner. This new representation can effectively convey the information of hundreds of high-resolution images in one compact format and allows photorealistic synthesis of novel views. In this work, using the variant of NeRF called Plenoxels, we create the first large-scale implicit representation datasets for perception tasks, called the PeRFception, which consists of two parts that incorporate both object-centric and scene-centric scans for classification and segmentation. It shows a significant memory compression rate (96.4\%) from the original dataset, while containing both 2D and 3D information in a unified form. We construct the classification and segmentation models that directly take as input this implicit format and also propose a novel augmentation technique to avoid overfitting on backgrounds of images. The code and data are publicly available in https://postech-cvlab.github.io/PeRFception .