Syn2Real: A New Benchmark forSynthetic-to-Real Visual Domain Adaptation

TL;DR

This paper introduces Syn2Real, a comprehensive benchmark for evaluating synthetic-to-real visual domain adaptation across multiple tasks, highlighting current challenges and gaps in performance.

Contribution

We present a large-scale, multi-task benchmark for synthetic-to-real domain adaptation, covering classification and detection, to facilitate progress in this research area.

Findings

Significant performance gap between closed-set and open-set tasks

Current methods perform well on simple classification but struggle with detection

Benchmark reveals need for more robust adaptation techniques

Abstract

Unsupervised transfer of object recognition models from synthetic to real data is an important problem with many potential applications. The challenge is how to "adapt" a model trained on simulated images so that it performs well on real-world data without any additional supervision. Unfortunately, current benchmarks for this problem are limited in size and task diversity. In this paper, we present a new large-scale benchmark called Syn2Real, which consists of a synthetic domain rendered from 3D object models and two real-image domains containing the same object categories. We define three related tasks on this benchmark: closed-set object classification, open-set object classification, and object detection. Our evaluation of multiple state-of-the-art methods reveals a large gap in adaptation performance between the easier closed-set classification task and the more difficult open-set and detection tasks. We conclude that developing adaptation methods that work well across all three tasks presents a significant future challenge for syn2real domain transfer.