Prototypical Progressive Alignment and Reweighting for Generalizable Semantic Segmentation

TL;DR

This paper introduces PPAR, a novel framework that uses CLIP-based prototypes, progressive alignment, and reweighting to improve the generalization of semantic segmentation models to unseen domains.

Contribution

The paper proposes a new method combining CLIP-derived prototypes, progressive alignment, and reweighting to enhance domain generalization in semantic segmentation.

Findings

PPAR achieves state-of-the-art results on multiple benchmarks.

The progressive alignment strategy effectively reduces domain gaps.

Prototypical reweighting mitigates negative transfer from irrelevant source data.

Abstract

Generalizable semantic segmentation aims to perform well on unseen target domains, a critical challenge due to real-world applications requiring high generalizability. Class-wise prototypes, representing class centroids, serve as domain-invariant cues that benefit generalization due to their stability and semantic consistency. However, this approach faces three challenges. First, existing methods often adopt coarse prototypical alignment strategies, which may hinder performance. Second, naive prototypes computed by averaging source batch features are prone to overfitting and may be negatively affected by unrelated source data. Third, most methods treat all source samples equally, ignoring the fact that different features have varying adaptation difficulties. To address these limitations, we propose a novel framework for generalizable semantic segmentation: Prototypical Progressive Alignment and Reweighting (PPAR), leveraging the strong generalization ability of the CLIP model. Specifically, we define two prototypes: the Original Text Prototype (OTP) and Visual Text Prototype (VTP), generated via CLIP to serve as a solid base for alignment. We then introduce a progressive alignment strategy that aligns features in an easy-to-difficult manner, reducing domain gaps gradually. Furthermore, we propose a prototypical reweighting mechanism that estimates the reliability of source data and adjusts its contribution, mitigating the effect of irrelevant or harmful features (i.e., reducing negative transfer). We also provide a theoretical analysis showing the alignment between our method and domain generalization theory. Extensive experiments across multiple benchmarks demonstrate that PPAR achieves state-of-the-art performance, validating its effectiveness.