Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask

TL;DR

This paper presents a method to generate multi-view consistent, photorealistic images of natural scenes from a single semantic mask using semantic fields and off-the-shelf models, trained on internet images.

Contribution

It introduces a novel approach that leverages semantic fields as intermediate representations for 3D-aware scene synthesis from a single input.

Findings

Outperforms baseline methods in realism and consistency

Produces multi-view videos of natural scenes

Works with only single-image training data

Abstract

We introduce a novel approach that takes a single semantic mask as input to synthesize multi-view consistent color images of natural scenes, trained with a collection of single images from the Internet. Prior works on 3D-aware image synthesis either require multi-view supervision or learning category-level prior for specific classes of objects, which can hardly work for natural scenes. Our key idea to solve this challenging problem is to use a semantic field as the intermediate representation, which is easier to reconstruct from an input semantic mask and then translate to a radiance field with the assistance of off-the-shelf semantic image synthesis models. Experiments show that our method outperforms baseline methods and produces photorealistic, multi-view consistent videos of a variety of natural scenes.