Undoing the Damage of Label Shift for Cross-domain Semantic Segmentation

TL;DR

This paper addresses the overlooked issue of label shift in cross-domain semantic segmentation, proposing a method to align conditional distributions and correct classifier bias, leading to significant performance improvements on urban scene benchmarks.

Contribution

It introduces a novel approach that explicitly tackles label shift in CDSS by class-level feature alignment and classifier bias correction, achieving state-of-the-art results.

Findings

Outperforms previous methods by a large margin

Achieves 59.3% mIoU on GTA5 to Cityscapes

Effectively corrects classifier bias caused by label shift

Abstract

Existing works typically treat cross-domain semantic segmentation (CDSS) as a data distribution mismatch problem and focus on aligning the marginal distribution or conditional distribution. However, the label shift issue is unfortunately overlooked, which actually commonly exists in the CDSS task, and often causes a classifier bias in the learnt model. In this paper, we give an in-depth analysis and show that the damage of label shift can be overcome by aligning the data conditional distribution and correcting the posterior probability. To this end, we propose a novel approach to undo the damage of the label shift problem in CDSS. In implementation, we adopt class-level feature alignment for conditional distribution alignment, as well as two simple yet effective methods to rectify the classifier bias from source to target by remolding the classifier predictions. We conduct extensive experiments on the benchmark datasets of urban scenes, including GTA5 to Cityscapes and SYNTHIA to Cityscapes, where our proposed approach outperforms previous methods by a large margin. For instance, our model equipped with a self-training strategy reaches 59.3% mIoU on GTA5 to Cityscapes, pushing to a new state-of-the-art. The code will be available at https://github.com/manmanjun/Undoing UDA.