Exploiting Semantics in Adversarial Training for Image-Level Domain Adaptation

TL;DR

This paper introduces a semantic-aware domain adaptation method using image translation GANs to improve real-world image segmentation performance trained on synthetic data.

Contribution

We propose a novel domain-to-domain image translation GAN with semantic constraints to reduce domain shift in semantic segmentation tasks.

Findings

Improved segmentation accuracy by over 16% mIoU on real images.

Semantic constraints help avoid artifacts in image translation.

Method effectively bridges the gap between synthetic and real images.

Abstract

Performance achievable by modern deep learning approaches are directly related to the amount of data used at training time. Unfortunately, the annotation process is notoriously tedious and expensive, especially for pixel-wise tasks like semantic segmentation. Recent works have proposed to rely on synthetically generated imagery to ease the training set creation. However, models trained on these kind of data usually under-perform on real images due to the well known issue of domain shift. We address this problem by learning a domain-to-domain image translation GAN to shrink the gap between real and synthetic images. Peculiarly to our method, we introduce semantic constraints into the generation process to both avoid artifacts and guide the synthesis. To prove the effectiveness of our proposal, we show how a semantic segmentation CNN trained on images from the synthetic GTA dataset adapted by our method can improve performance by more than 16% mIoU with respect to the same model trained on synthetic images.