Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fields

TL;DR

Sem2NeRF introduces a novel method to reconstruct 3D scenes from single-view semantic masks using NeRF, leveraging region-aware learning to improve accuracy and outperform baselines.

Contribution

The paper proposes Sem2NeRF, a new framework that encodes semantic masks into 3D scene representations conditioned on a pre-trained decoder, with a novel region-aware learning strategy.

Findings

Outperforms several strong baselines on benchmark datasets.

Effectively encodes semantic masks into 3D NeRF representations.

Demonstrates the feasibility of single-view semantic-to-NeRF translation.

Abstract

Image translation and manipulation have gain increasing attention along with the rapid development of deep generative models. Although existing approaches have brought impressive results, they mainly operated in 2D space. In light of recent advances in NeRF-based 3D-aware generative models, we introduce a new task, Semantic-to-NeRF translation, that aims to reconstruct a 3D scene modelled by NeRF, conditioned on one single-view semantic mask as input. To kick-off this novel task, we propose the Sem2NeRF framework. In particular, Sem2NeRF addresses the highly challenging task by encoding the semantic mask into the latent code that controls the 3D scene representation of a pre-trained decoder. To further improve the accuracy of the mapping, we integrate a new region-aware learning strategy into the design of both the encoder and the decoder. We verify the efficacy of the proposed Sem2NeRF and demonstrate that it outperforms several strong baselines on two benchmark datasets. Code and video are available at https://donydchen.github.io/sem2nerf/