Adversarial Semantic Hallucination for Domain Generalized Semantic Segmentation

TL;DR

This paper introduces ASH, an adversarial semantic hallucination method that enhances domain generalization in semantic segmentation by generating class-specific stylized images without target domain data.

Contribution

The paper proposes a novel class-conditioned adversarial hallucination approach that considers class-wise differences to improve domain generalization in semantic segmentation.

Findings

Competitive performance on Cityscapes and Mapillary datasets.

Effective class-wise style augmentation improves segmentation robustness.

Outperforms some existing domain generalization methods.

Abstract

Convolutional neural networks typically perform poorly when the test (target domain) and training (source domain) data have significantly different distributions. While this problem can be mitigated by using the target domain data to align the source and target domain feature representations, the target domain data may be unavailable due to privacy concerns. Consequently, there is a need for methods that generalize well despite restricted access to target domain data during training. In this work, we propose an adversarial semantic hallucination approach (ASH), which combines a class-conditioned hallucination module and a semantic segmentation module. Since the segmentation performance varies across different classes, we design a semantic-conditioned style hallucination module to generate affine transformation parameters from semantic information in the segmentation probability maps of the source domain image. Unlike previous adaptation approaches, which treat all classes equally, ASH considers the class-wise differences. The segmentation module and the hallucination module compete adversarially, with the hallucination module generating increasingly "difficult" stylized images to challenge the segmentation module. In response, the segmentation module improves as it is trained with generated samples at an appropriate class-wise difficulty level. Our results on the Cityscapes and Mapillary benchmark datasets show that our method is competitive with state of the art work. Code is made available at https://github.com/gabriel-tjio/ASH.