In defense of the two-stage framework for open-set domain adaptive semantic segmentation

TL;DR

This paper introduces SATS, a two-stage training framework for open-set domain adaptive semantic segmentation that improves the separation and adaptation of known and unknown classes, leading to better performance on benchmarks.

Contribution

The paper proposes a novel two-step approach, separating known/unknown classes before domain adaptation, and introduces hard unknown exploration for enhanced unknown class learning.

Findings

Achieves +3.85% H-Score on GTA5-to-Cityscapes

Achieves +18.64% H-Score on SYNTHIA-to-Cityscapes

Outperforms previous state-of-the-art methods

Abstract

Open-Set Domain Adaptation for Semantic Segmentation (OSDA-SS) presents a significant challenge, as it requires both domain adaptation for known classes and the distinction of unknowns. Existing methods attempt to address both tasks within a single unified stage. We question this design, as the annotation imbalance between known and unknown classes often leads to negative transfer of known classes and underfitting for unknowns. To overcome these issues, we propose SATS, a Separating-then-Adapting Training Strategy, which addresses OSDA-SS through two sequential steps: known/unknown separation and unknown-aware domain adaptation. By providing the model with more accurate and well-aligned unknown classes, our method ensures a balanced learning of discriminative features for both known and unknown classes, steering the model toward discovering truly unknown objects. Additionally, we present hard unknown exploration, an innovative data augmentation method that exposes the model to more challenging unknowns, strengthening its ability to capture more comprehensive understanding of target unknowns. We evaluate our method on public OSDA-SS benchmarks. Experimental results demonstrate that our method achieves a substantial advancement, with a +3.85% H-Score improvement for GTA5-to-Cityscapes and +18.64% for SYNTHIA-to-Cityscapes, outperforming previous state-of-the-art methods.