GAN2X: Non-Lambertian Inverse Rendering of Image GANs

TL;DR

GAN2X introduces an unsupervised method for inverse rendering that recovers 3D shape, material, and illumination from single 2D images without paired data, advancing 3D understanding and editing capabilities.

Contribution

It is the first to recover non-Lambertian material properties using unpaired images and pseudo data generated by GANs, with a specularity-aware neural surface representation.

Findings

Accurately decomposes images into 3D shape, albedo, and specular properties.

Achieves state-of-the-art unsupervised 3D face reconstruction.

Enables applications like image editing and 3D GAN lifting.

Abstract

2D images are observations of the 3D physical world depicted with the geometry, material, and illumination components. Recovering these underlying intrinsic components from 2D images, also known as inverse rendering, usually requires a supervised setting with paired images collected from multiple viewpoints and lighting conditions, which is resource-demanding. In this work, we present GAN2X, a new method for unsupervised inverse rendering that only uses unpaired images for training. Unlike previous Shape-from-GAN approaches that mainly focus on 3D shapes, we take the first attempt to also recover non-Lambertian material properties by exploiting the pseudo paired data generated by a GAN. To achieve precise inverse rendering, we devise a specularity-aware neural surface representation that continuously models the geometry and material properties. A shading-based refinement technique is adopted to further distill information in the target image and recover more fine details. Experiments demonstrate that GAN2X can accurately decompose 2D images to 3D shape, albedo, and specular properties for different object categories, and achieves the state-of-the-art performance for unsupervised single-view 3D face reconstruction. We also show its applications in downstream tasks including real image editing and lifting 2D GANs to decomposed 3D GANs.