Open-Vocabulary Domain Generalization in Urban-Scene Segmentation

TL;DR

This paper introduces a new benchmark and method for open-vocabulary domain generalization in urban-scene segmentation, addressing the challenge of recognizing unseen categories across diverse unseen environments.

Contribution

It proposes the first benchmark for OVDG-SS in autonomous driving and introduces S2-Corr, a novel correlation refinement mechanism to improve robustness across domains.

Findings

S2-Corr improves cross-domain segmentation accuracy.

The benchmark covers synthetic-to-real and real-to-real generalization.

The proposed method outperforms existing approaches in efficiency and accuracy.

Abstract

Domain Generalization in Semantic Segmentation (DG-SS) aims to enable segmentation models to perform robustly in unseen environments. However, conventional DG-SS methods are restricted to a fixed set of known categories, limiting their applicability in open-world scenarios. Recent progress in Vision-Language Models (VLMs) has advanced Open-Vocabulary Semantic Segmentation (OV-SS) by enabling models to recognize a broader range of concepts. Yet, these models remain sensitive to domain shifts and struggle to maintain robustness when deployed in unseen environments, a challenge that is particularly severe in urban-driving scenarios. To bridge this gap, we introduce Open-Vocabulary Domain Generalization in Semantic Segmentation (OVDG-SS), a new setting that jointly addresses unseen domains and unseen categories. We introduce the first benchmark for OVDG-SS in autonomous driving, addressing a previously unexplored problem and covering both synthetic-to-real and real-to-real generalization across diverse unseen domains and unseen categories. In OVDG-SS, we observe that domain shifts often distort text-image correlations in pre-trained VLMs, which hinders the performance of OV-SS models. To tackle this challenge, we propose S2-Corr, a state-space-driven text-image correlation refinement mechanism that mitigates domain-induced distortions and produces more consistent text-image correlations under distribution changes. Extensive experiments on our constructed benchmark demonstrate that the proposed method achieves superior cross-domain performance and efficiency compared to existing OV-SS approaches.