On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset

TL;DR

This paper introduces a new large-scale dataset of real and simulated 3D object images with multiple factors of variation, aiming to evaluate and improve the transfer of disentangled representations from simulation to real-world data.

Contribution

The authors present a novel dataset with real and simulated images for disentanglement research and analyze how well models trained on simulation transfer to real data.

Findings

Models transfer poorly from simulation to real data.

Model and hyperparameter tuning improve transfer effectiveness.

The dataset enables systematic evaluation of transferability in disentanglement.

Abstract

Learning meaningful and compact representations with disentangled semantic aspects is considered to be of key importance in representation learning. Since real-world data is notoriously costly to collect, many recent state-of-the-art disentanglement models have heavily relied on synthetic toy data-sets. In this paper, we propose a novel data-set which consists of over one million images of physical 3D objects with seven factors of variation, such as object color, shape, size and position. In order to be able to control all the factors of variation precisely, we built an experimental platform where the objects are being moved by a robotic arm. In addition, we provide two more datasets which consist of simulations of the experimental setup. These datasets provide for the first time the possibility to systematically investigate how well different disentanglement methods perform on real data in comparison to simulation, and how simulated data can be leveraged to build better representations of the real world. We provide a first experimental study of these questions and our results indicate that learned models transfer poorly, but that model and hyperparameter selection is an effective means of transferring information to the real world.