Unsupervised Generative 3D Shape Learning from Natural Images

TL;DR

This paper introduces a fully unsupervised method for learning 3D shape generative models directly from natural images, without relying on 3D annotations, stereo videos, or ego-motion data.

Contribution

It presents a novel two-stage GAN-based approach that learns 3D representations from 2D images, disentangling shape from viewpoint, using a differentiable renderer.

Findings

Successfully learned 3D face shapes from natural images

Achieved realistic 3D shape generation without 3D supervision

Demonstrated disentangled 3D representation learning

Abstract

In this paper we present, to the best of our knowledge, the first method to learn a generative model of 3D shapes from natural images in a fully unsupervised way. For example, we do not use any ground truth 3D or 2D annotations, stereo video, and ego-motion during the training. Our approach follows the general strategy of Generative Adversarial Networks, where an image generator network learns to create image samples that are realistic enough to fool a discriminator network into believing that they are natural images. In contrast, in our approach the image generation is split into 2 stages. In the first stage a generator network outputs 3D objects. In the second, a differentiable renderer produces an image of the 3D objects from random viewpoints. The key observation is that a realistic 3D object should yield a realistic rendering from any plausible viewpoint. Thus, by randomizing the choice of the viewpoint our proposed training forces the generator network to learn an interpretable 3D representation disentangled from the viewpoint. In this work, a 3D representation consists of a triangle mesh and a texture map that is used to color the triangle surface by using the UV-mapping technique. We provide analysis of our learning approach, expose its ambiguities and show how to overcome them. Experimentally, we demonstrate that our method can learn realistic 3D shapes of faces by using only the natural images of the FFHQ dataset.