Adversarial Inverse Graphics Networks: Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision

TL;DR

This paper introduces Adversarial Inverse Graphics Networks (AIGNs), a weakly supervised framework that learns 2D-to-3D lifting and image translation from unpaired data by combining rendering feedback with distribution matching.

Contribution

The paper presents a novel weakly supervised neural network model that integrates rendering feedback and distribution matching to learn complex image-to-structure mappings without paired annotations.

Findings

Outperforms supervised models in 3D pose and structure estimation

Successfully applies to facial transformation with biases

Handles unpaired data effectively

Abstract

Researchers have developed excellent feed-forward models that learn to map images to desired outputs, such as to the images' latent factors, or to other images, using supervised learning. Learning such mappings from unlabelled data, or improving upon supervised models by exploiting unlabelled data, remains elusive. We argue that there are two important parts to learning without annotations: (i) matching the predictions to the input observations, and (ii) matching the predictions to known priors. We propose Adversarial Inverse Graphics networks (AIGNs): weakly supervised neural network models that combine feedback from rendering their predictions, with distribution matching between their predictions and a collection of ground-truth factors. We apply AIGNs to 3D human pose estimation and 3D structure and egomotion estimation, and outperform models supervised by only paired annotations. We further apply AIGNs to facial image transformation using super-resolution and inpainting renderers, while deliberately adding biases in the ground-truth datasets. Our model seamlessly incorporates such biases, rendering input faces towards young, old, feminine, masculine or Tom Cruise-like equivalents (depending on the chosen bias), or adding lip and nose augmentations while inpainting concealed lips and noses.

Full text

Supplementary Material for

“Adversarial Inverse Graphics Networks: Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision”

Hsiao-Yu Fish Tung   Adam W. Harley 11footnotemark: 1   William Seto 11footnotemark: 1   Katerina Fragkiadaki

Carnegie Mellon University

{htung,aharley,wseto,katef}@cs.cmu.edu equal contribution

1 Parametric vs. non-parametric decoders

Here we discuss the benefits of using non-parametric and domain-specific renderers, over learned decoders. Both the proposed model and CycleGAN [CycleGAN2017] can be viewed as autoencoders: the input is first transformed into a target domain, and then transformed back to its original space. A parametric decoder could be more desirable, for the reason that we do not need to hand-engineer a mapping function from the target domain back to the inputs. However, simply using reconstruction loss and adversarial loss does not guarantee that the predictions look spatially similar to the inputs. In tasks such as image-to-image translation, spatial precision can be of critical importance. With a parametric decoder, the transformed input can be viewed as a information bottleneck, and as long as the decoder can correctly “guess” the final output from the transformed input (i.e., the code), the code is valid and the solution is optimal.

To support this point, we conduct an experiment on image inpainting using the MNIST dataset. Similar to the parametric encoder-decoder described in the main text, the network has two main parts: (1) an encoder that transforms the input (a partially obscured image of a digit) into prediction (a hallucinated digit), and (2) a decoder that transforms the prediction back into the input. Instead of using convolutional layers, which have an architectural bias on preserving spatial relationships, we use fully-connected layers in both the encoder and the decoder. This is important, because such architectural conveniences are unavailable in less-structured tasks, such as 3D pose prediction and SfM. We train the model with a reconstruction loss on the decoder, and adversarial loss on the encoder.

The results are shown in Figure 1. While inpainting, the encoder (incorrectly) transforms many of the digits into other digits. For instance, several obscured “1” images are inpainted as “4”. In the parametric decoding process, however, these errors are undone, and the original input is recovered successfully. In other words, the decoder takes the burden of the reconstruction loss, allowing the encoder to learn an inaccurate latent space. Parameter-free rendering avoids this problem.

2 Additional experiments and details

In the sections to follow, we provide implementation details, including architecture descriptions for the generator and discriminator in each task, and training details. Additionally, we provide more experimental results.

2.1 3D human pose estimation from static images

Figure 2 shows the architecture of our generator network for 3D human pose estimation from a single RGB image. Our generator predicts weights over the shape bases $\alpha$, rotation $R,$ translation $T$ and focal length $f$, as described in our paper. The generator takes as input a set of 2D body joint heatmaps. We use convolutional pose machines [wei2016cpm] to estimate 2D keypoints, and convert them into heatmaps by creating a Gaussian distribution centered around each 2D keypoint. The network consists of 8 convolutional layers with leaky ReLU activations and batch normalization and two fully connected layers at the end that map to the desired outputs. The width, height and number of channels for each layer is specified in Figure 6. The discriminator for this task consists of five fully connected layers with featuremap depth 512, 512, 256, 256 and 1, with a leaky ReLU and batch normalization after each layer. The discriminator takes all values output from the generator (i.e., $\alpha$, $R,$ $T$, $f$) as input.

In all experiments, we set the variance for the Gaussian heatmap $\sigma$ to 0.25, and the dimensionality of our PCA shape basis to 60 (out of 96 total bases). The dimensionality reduction is small, and indeed, we only use basis weights for ease of prediction, relying on our adversarial priors (rather than PCA) to regularize the 3D shape prediction. We use gradient descent for both generator and discriminator training. Learning rate for reconstruction loss is set to 0.00001 and learning rate for the adversarial loss is set to 0.0001. All parameters are initialized with random sampling from zero mean normal distributions with variance of 0.02.

In Figure 3, we show predicted 3D human poses on images from the MPII dataset [andriluka14cvpr] using the ground-truth 2D keypoints available. Our model generalizes well on unseen images without any further self-supervised finetuning, though we would expect additional self-supervised finetuning to further improve performance.

2.2 Structure from Motion

Our generator networks for the task of structure from motion is illustrated in Figure 4. It includes three encoder-decoder convolutional networks with skip connections, which solve for optical flow, depth, and camera motion. The egomotion network uses RGB, optical flow and an angle field as input, and estimates the camera motion in SE(3). The depth network takes an RGB image as input and predicts logdepth. The depth discriminator consists of four convolution layers with batch normalization on the second and the third layers, and leaky ReLU activation after each layer. The depth discriminator is fully convolutional as we are interested in the realism of every depth patch, as opposed to the depthmap as a whole.

The egomotion discriminator is a 3-layer fully-connected network that takes $\{R,T\}$ matrices as input. The hidden layers have 128, 128, and 64 neurons, respectively, with batch normalization and a leaky ReLU after each layer.

Stabilizing training. In order to make sure that generators and discriminators progress together during training, we update the generator only when the discriminator has low enough loss. We add an updating heuristic such that if the likelihood loss of the discriminator is above a threshold $\theta_{d},$ we do not update the generator. While discriminator is strong enough (below this threshold) and the generator is relatively weak (below a different threshold $\theta_{g}$), we update the generator twice in the iteration. In the experiments, we set $\theta_{d}$ to 0.695 and $\theta_{g}$ to 0.75.

2.3 Image Super-Resolution

In Figure 6, we show the architecture of the generator and discriminator for image super-resolution. The input image is first passed through a convolutional layer with 64 channels, then $n$ “residual blocks”. Each residual block consists of two convolutional layers, with a batch normalization after each convolution layer and ReLU activation after the first batch normalization. The output from the last block is further passed to two deconvolution layers and generates the final image. The discriminator for this task consists of five convolution layers that use leaky ReLU activations and batch normalization, and one fully-connected layer that outputs one final value. In all experiments, we use Adam optimizer with learning rate 0.0001.All parameters are initialized with truncated normal distribution with variance 0.02.

In Figures 8, 9 and 10, we compare our model with Attribute2Image [DBLP:journals/corr/YanYSL15] and with Unsupervised Image Translation [DBLP:journals/corr/DongNWG17] for gender and age transformations. We use the code provided by the authors for our comparisons. In Figures 5 and 7, we show additional results of our model on gender and age transformation.

2.4 Inpainting

Figure 6 illustrates the architecture of our generator and discriminator for image inpainting. The occluded input image and the corresponding mask are separately passed through two convolution layers, and then concatenated. The concatenated outputs are then passed to three deconvolutional layers to generate the inpainted image. The discriminator consists of four convolutional layers with leaky ReLU and batch normalization layers, and one fully connected layer that outputs one final value. In all experiments, we use the Adam optimizer, with a learning rate $1\mathrm{e}{-4}$All parameters are initialized from the truncated Normal distribution, with variance 0.02.

In Figure 11, we show additional results on biased inpainting for making bigger lips.