Keep it Unreal: Bridging the Realism Gap for 2.5D Recognition with Geometry Priors Only

TL;DR

This paper introduces a CAD-model-based generative adversarial network that enhances synthetic depth images to resemble real sensor data, improving recognition tasks without needing real scans for training.

Contribution

It presents a novel unsupervised method that uses only CAD models and synthetic data to bridge the realism gap in depth-based recognition.

Findings

Improves recognition accuracy on real data using synthetic training.

Achieves results comparable to models trained on real scans.

Simplifies training pipelines by removing the need for real data.

Abstract

With the increasing availability of large databases of 3D CAD models, depth-based recognition methods can be trained on an uncountable number of synthetically rendered images. However, discrepancies with the real data acquired from various depth sensors still noticeably impede progress. Previous works adopted unsupervised approaches to generate more realistic depth data, but they all require real scans for training, even if unlabeled. This still represents a strong requirement, especially when considering real-life/industrial settings where real training images are hard or impossible to acquire, but texture-less 3D models are available. We thus propose a novel approach leveraging only CAD models to bridge the realism gap. Purely trained on synthetic data, playing against an extensive augmentation pipeline in an unsupervised manner, our generative adversarial network learns to effectively segment depth images and recover the clean synthetic-looking depth information even from partial occlusions. As our solution is not only fully decoupled from the real domains but also from the task-specific analytics, the pre-processed scans can be handed to any kind and number of recognition methods also trained on synthetic data. Through various experiments, we demonstrate how this simplifies their training and consistently enhances their performance, with results on par with the same methods trained on real data, and better than usual approaches doing the reverse mapping.