Sensor Transfer: Learning Optimal Sensor Effect Image Augmentation for Sim-to-Real Domain Adaptation

TL;DR

This paper introduces a learned augmentation network that applies sensor-specific effects to synthetic images, reducing the domain gap and improving object detection performance in urban driving scenarios.

Contribution

The work presents a novel physically-based augmentation pipeline that models sensor effects to enhance synthetic-to-real domain adaptation.

Findings

Augmenting synthetic data with sensor effects improves detection accuracy.

The proposed method reduces domain gap between synthetic and real datasets.

Sensor effect augmentation outperforms traditional augmentation techniques.

Abstract

Performance on benchmark datasets has drastically improved with advances in deep learning. Still, cross-dataset generalization performance remains relatively low due to the domain shift that can occur between two different datasets. This domain shift is especially exaggerated between synthetic and real datasets. Significant research has been done to reduce this gap, specifically via modeling variation in the spatial layout of a scene, such as occlusions, and scene environmental factors, such as time of day and weather effects. However, few works have addressed modeling the variation in the sensor domain as a means of reducing the synthetic to real domain gap. The camera or sensor used to capture a dataset introduces artifacts into the image data that are unique to the sensor model, suggesting that sensor effects may also contribute to domain shift. To address this, we propose a learned augmentation network composed of physically-based augmentation functions. Our proposed augmentation pipeline transfers specific effects of the sensor model -- chromatic aberration, blur, exposure, noise, and color temperature -- from a real dataset to a synthetic dataset. We provide experiments that demonstrate that augmenting synthetic training datasets with the proposed learned augmentation framework reduces the domain gap between synthetic and real domains for object detection in urban driving scenes.