Intra- & Extra-Source Exemplar-Based Style Synthesis for Improved Domain Generalization

TL;DR

This paper introduces an exemplar-based style synthesis method using a masked noise encoder for StyleGAN2 to enhance domain generalization in semantic segmentation, significantly improving performance across various data shifts.

Contribution

It proposes a novel masked noise encoder for StyleGAN2 inversion and a style augmentation technique called ISSA that boosts domain generalization in semantic segmentation tasks.

Findings

Up to 12.4% mIoU improvement on driving-scene segmentation under data shifts.

ISSA is model-agnostic and enhances CNNs and Transformers.

The method improves state-of-the-art solutions like RobustNet by 3% mIoU.

Abstract

The generalization with respect to domain shifts, as they frequently appear in applications such as autonomous driving, is one of the remaining big challenges for deep learning models. Therefore, we propose an exemplar-based style synthesis pipeline to improve domain generalization in semantic segmentation. Our method is based on a novel masked noise encoder for StyleGAN2 inversion. The model learns to faithfully reconstruct the image, preserving its semantic layout through noise prediction. Using the proposed masked noise encoder to randomize style and content combinations in the training set, i.e., intra-source style augmentation (ISSA) effectively increases the diversity of training data and reduces spurious correlation. As a result, we achieve up to $12.4\%$ mIoU improvements on driving-scene semantic segmentation under different types of data shifts, i.e., changing geographic locations, adverse weather conditions, and day to night. ISSA is model-agnostic and straightforwardly applicable with CNNs and Transformers. It is also complementary to other domain generalization techniques, e.g., it improves the recent state-of-the-art solution RobustNet by $3\%$ mIoU in Cityscapes to Dark Z\"urich. In addition, we demonstrate the strong plug-n-play ability of the proposed style synthesis pipeline, which is readily usable for extra-source exemplars e.g., web-crawled images, without any retraining or fine-tuning. Moreover, we study a new use case to indicate neural network's generalization capability by building a stylized proxy validation set. This application has significant practical sense for selecting models to be deployed in the open-world environment. Our code is available at \url{https://github.com/boschresearch/ISSA}.