Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask
Shangzan Zhang, Sida Peng, Tianrun Chen, Linzhan Mou, Haotong Lin,, Kaicheng Yu, Yiyi Liao, Xiaowei Zhou

TL;DR
This paper presents a method to generate multi-view consistent, photorealistic images of natural scenes from a single semantic mask using semantic fields and off-the-shelf models, trained on internet images.
Contribution
It introduces a novel approach that leverages semantic fields as intermediate representations for 3D-aware scene synthesis from a single input.
Findings
Outperforms baseline methods in realism and consistency
Produces multi-view videos of natural scenes
Works with only single-image training data
Abstract
We introduce a novel approach that takes a single semantic mask as input to synthesize multi-view consistent color images of natural scenes, trained with a collection of single images from the Internet. Prior works on 3D-aware image synthesis either require multi-view supervision or learning category-level prior for specific classes of objects, which can hardly work for natural scenes. Our key idea to solve this challenging problem is to use a semantic field as the intermediate representation, which is easier to reconstruct from an input semantic mask and then translate to a radiance field with the assistance of off-the-shelf semantic image synthesis models. Experiments show that our method outperforms baseline methods and produces photorealistic, multi-view consistent videos of a variety of natural scenes.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Vision and Imaging · Computer Graphics and Visualization Techniques · Generative Adversarial Networks and Image Synthesis
