Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs

TL;DR

This paper demonstrates that pre-trained 2D GANs implicitly contain 3D shape information, enabling unsupervised 3D shape reconstruction from single images without requiring explicit 3D annotations.

Contribution

It introduces a novel framework that mines 3D geometric cues from off-the-shelf 2D GANs to recover 3D shapes in an unsupervised manner, without strong shape assumptions.

Findings

Successfully recovers 3D shapes for faces, cats, cars, and buildings

Enables high-quality relighting and object rotation from single images

Outperforms previous methods in 3D shape reconstruction and face rotation

Abstract

Natural images are projections of 3D objects on a 2D image plane. While state-of-the-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the underlying 3D object structures. And if so, how could we exploit such knowledge to recover the 3D shapes of objects in the images? To answer these questions, in this work, we present the first attempt to directly mine 3D geometric cues from an off-the-shelf 2D GAN that is trained on RGB images only. Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner. The core of our framework is an iterative strategy that explores and exploits diverse viewpoint and lighting variations in the GAN image manifold. The framework does not require 2D keypoint or 3D annotations, or strong assumptions on object shapes (e.g. shapes are symmetric), yet it successfully recovers 3D shapes with high precision for human faces, cats, cars, and buildings. The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation. We quantitatively demonstrate the effectiveness of our approach compared to previous methods in both 3D shape reconstruction and face rotation. Our code is available at https://github.com/XingangPan/GAN2Shape.