3D-aware Conditional Image Synthesis

TL;DR

This paper introduces pix2pix3D, a 3D-aware generative model that synthesizes photorealistic images from label maps with controllable viewpoints, integrating neural radiance fields for explicit 3D control.

Contribution

It extends conditional generative models with neural radiance fields to enable 3D-aware image synthesis from 2D label maps.

Findings

Achieves controllable viewpoint synthesis from label maps.

Learns to assign labels, color, and density to 3D points.

Supports interactive editing of label maps from any viewpoint.

Abstract

We propose pix2pix3D, a 3D-aware conditional generative model for controllable photorealistic image synthesis. Given a 2D label map, such as a segmentation or edge map, our model learns to synthesize a corresponding image from different viewpoints. To enable explicit 3D user control, we extend conditional generative models with neural radiance fields. Given widely-available monocular images and label map pairs, our model learns to assign a label to every 3D point in addition to color and density, which enables it to render the image and pixel-aligned label map simultaneously. Finally, we build an interactive system that allows users to edit the label map from any viewpoint and generate outputs accordingly.