Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation

TL;DR

Panoptic Neural Fields (PNF) is a neural scene representation that decomposes scenes into objects and background, enabling tasks like view synthesis, segmentation, and editing from color images.

Contribution

Introduces PNF, an object-aware neural scene model that efficiently represents scenes with object-specific MLPs and background, leveraging meta-learning and self-supervision.

Findings

Effective for novel view synthesis

Accurate 2D panoptic segmentation

Supports 3D scene editing and depth prediction

Abstract

We present Panoptic Neural Fields (PNF), an object-aware neural scene representation that decomposes a scene into a set of objects (things) and background (stuff). Each object is represented by an oriented 3D bounding box and a multi-layer perceptron (MLP) that takes position, direction, and time and outputs density and radiance. The background stuff is represented by a similar MLP that additionally outputs semantic labels. Each object MLPs are instance-specific and thus can be smaller and faster than previous object-aware approaches, while still leveraging category-specific priors incorporated via meta-learned initialization. Our model builds a panoptic radiance field representation of any scene from just color images. We use off-the-shelf algorithms to predict camera poses, object tracks, and 2D image semantic segmentations. Then we jointly optimize the MLP weights and bounding box parameters using analysis-by-synthesis with self-supervision from color images and pseudo-supervision from predicted semantic segmentations. During experiments with real-world dynamic scenes, we find that our model can be used effectively for several tasks like novel view synthesis, 2D panoptic segmentation, 3D scene editing, and multiview depth prediction.