Beyond Photo Realism for Domain Adaptation from Synthetic Data

TL;DR

This paper evaluates various synthetic data generation techniques for training deep models, introduces a novel learned synthesis method that outperforms traditional graphical methods, and provides insights into domain adaptation from synthetic to real data.

Contribution

It presents a new learned synthesis technique for generating synthetic images, surpassing state-of-the-art graphical methods while reducing computational costs.

Findings

Learned synthesis improves classifier performance over traditional methods.

A new dataset enables comparison of real and synthetic scenes.

Ensemble GANs approach real data performance in domain adaptation.

Abstract

As synthetic imagery is used more frequently in training deep models, it is important to understand how different synthesis techniques impact the performance of such models. In this work, we perform a thorough evaluation of the effectiveness of several different synthesis techniques and their impact on the complexity of classifier domain adaptation to the "real" underlying data distribution that they seek to replicate. In addition, we propose a novel learned synthesis technique to better train classifier models than state-of-the-art offline graphical methods, while using significantly less computational resources. We accomplish this by learning a generative model to perform shading of synthetic geometry conditioned on a "g-buffer" representation of the scene to render, as well as a low sample Monte Carlo rendered image. The major contributions are (i) a dataset that allows comparison of real and synthetic versions of the same scene, (ii) an augmented data representation that boosts the stability of learning and improves the datasets accuracy, (iii) three different partially differentiable rendering techniques where lighting, denoising and shading are learned, and (iv) we improve a state of the art generative adversarial network (GAN) approach by using an ensemble of trained models to generate datasets that approach the performance of training on real data and surpass the performance of the full global illumination rendering.