Exploiting Domain Properties in Language-Driven Domain Generalization for Semantic Segmentation

TL;DR

This paper introduces DPMFormer, a novel framework for semantic segmentation that leverages domain-aware prompt learning, contrastive learning, and consistency training to improve domain generalization, achieving state-of-the-art results.

Contribution

The paper proposes a new domain generalization framework that integrates domain-aware prompts, contrastive learning, and robustness strategies for semantic segmentation.

Findings

Achieves state-of-the-art performance on DGSS benchmarks.

Demonstrates effectiveness of domain-aware prompts and contrastive learning.

Shows improved robustness against environmental variations.

Abstract

Recent domain generalized semantic segmentation (DGSS) studies have achieved notable improvements by distilling semantic knowledge from Vision-Language Models (VLMs). However, they overlook the semantic misalignment between visual and textual contexts, which arises due to the rigidity of a fixed context prompt learned on a single source domain. To this end, we present a novel domain generalization framework for semantic segmentation, namely Domain-aware Prompt-driven Masked Transformer (DPMFormer). Firstly, we introduce domain-aware prompt learning to facilitate semantic alignment between visual and textual cues. To capture various domain-specific properties with a single source dataset, we propose domain-aware contrastive learning along with the texture perturbation that diversifies the observable domains. Lastly, to establish a framework resilient against diverse environmental changes, we have proposed the domain-robust consistency learning which guides the model to minimize discrepancies of prediction from original and the augmented images. Through experiments and analyses, we demonstrate the superiority of the proposed framework, which establishes a new state-of-the-art on various DGSS benchmarks. The code is available at https://github.com/jone1222/DPMFormer.