Beyond RGB: Scene-Property Synthesis with Neural Radiance Fields

TL;DR

This paper introduces SS-NeRF, a neural rendering framework that synthesizes both realistic images and detailed scene properties, advancing 3D scene understanding through a unified generative approach.

Contribution

It presents a novel scene-property synthesis method based on NeRFs, enabling simultaneous rendering of images and scene attributes for improved scene understanding tasks.

Findings

Supports rendering of photo-realistic images from new viewpoints

Accurately synthesizes scene properties like geometry and semantics

Facilitates multiple scene understanding tasks within a single framework

Abstract

Comprehensive 3D scene understanding, both geometrically and semantically, is important for real-world applications such as robot perception. Most of the existing work has focused on developing data-driven discriminative models for scene understanding. This paper provides a new approach to scene understanding, from a synthesis model perspective, by leveraging the recent progress on implicit 3D representation and neural rendering. Building upon the great success of Neural Radiance Fields (NeRFs), we introduce Scene-Property Synthesis with NeRF (SS-NeRF) that is able to not only render photo-realistic RGB images from novel viewpoints, but also render various accurate scene properties (e.g., appearance, geometry, and semantics). By doing so, we facilitate addressing a variety of scene understanding tasks under a unified framework, including semantic segmentation, surface normal estimation, reshading, keypoint detection, and edge detection. Our SS-NeRF framework can be a powerful tool for bridging generative learning and discriminative learning, and thus be beneficial to the investigation of a wide range of interesting problems, such as studying task relationships within a synthesis paradigm, transferring knowledge to novel tasks, facilitating downstream discriminative tasks as ways of data augmentation, and serving as auto-labeller for data creation.