Autoencoder for Synthetic to Real Generalization: From Simple to More Complex Scenes

TL;DR

This paper proposes an autoencoder-based approach to improve the generalization from synthetic to real images, especially in complex scenes, by using semantic matching sampling techniques to enhance latent space invariance.

Contribution

It introduces a novel sampling method that enhances autoencoder generalization from synthetic to real images, outperforming fine-tuned classifiers on complex scenes.

Findings

Pre-trained feature extractors work well on simple scenes.

Semantic matching sampling improves real-world generalization.

The approach outperforms fine-tuned classification models on complex data.

Abstract

Learning on synthetic data and transferring the resulting properties to their real counterparts is an important challenge for reducing costs and increasing safety in machine learning. In this work, we focus on autoencoder architectures and aim at learning latent space representations that are invariant to inductive biases caused by the domain shift between simulated and real images showing the same scenario. We train on synthetic images only, present approaches to increase generalizability and improve the preservation of the semantics to real datasets of increasing visual complexity. We show that pre-trained feature extractors (e.g. VGG) can be sufficient for generalization on images of lower complexity, but additional improvements are required for visually more complex scenes. To this end, we demonstrate a new sampling technique, which matches semantically important parts of the image, while randomizing the other parts, leads to salient feature extraction and a neglection of unimportant parts. This helps the generalization to real data and we further show that our approach outperforms fine-tuned classification models.