Zero Shot Domain Adaptive Semantic Segmentation by Synthetic Data Generation and Progressive Adaptation

TL;DR

This paper introduces SDGPA, a novel approach for zero-shot domain adaptive semantic segmentation that uses synthetic data generated from text descriptions and progressive adaptation strategies, achieving state-of-the-art results.

Contribution

The paper proposes a new method combining synthetic data generation from text-to-image models with progressive adaptation and spatial editing to handle zero-shot domain shifts in semantic segmentation.

Findings

Achieves state-of-the-art performance in zero-shot semantic segmentation.

Effective synthetic data generation from text descriptions improves domain adaptation.

Progressive adaptation enhances training stability and robustness.

Abstract

Deep learning-based semantic segmentation models achieve impressive results yet remain limited in handling distribution shifts between training and test data. In this paper, we present SDGPA (Synthetic Data Generation and Progressive Adaptation), a novel method that tackles zero-shot domain adaptive semantic segmentation, in which no target images are available, but only a text description of the target domain's style is provided. To compensate for the lack of target domain training data, we utilize a pretrained off-the-shelf text-to-image diffusion model, which generates training images by transferring source domain images to target style. Directly editing source domain images introduces noise that harms segmentation because the layout of source images cannot be precisely maintained. To address inaccurate layouts in synthetic data, we propose a method that crops the source image, edits small patches individually, and then merges them back together, which helps improve spatial precision. Recognizing the large domain gap, SDGPA constructs an augmented intermediate domain, leveraging easier adaptation subtasks to enable more stable model adaptation to the target domain. Additionally, to mitigate the impact of noise in synthetic data, we design a progressive adaptation strategy, ensuring robust learning throughout the training process. Extensive experiments demonstrate that our method achieves state-of-the-art performance in zero-shot semantic segmentation. The code is available at https://github.com/ROUJINN/SDGPA