Leveraging Depth and Language for Open-Vocabulary Domain-Generalized Semantic Segmentation

TL;DR

This paper introduces Vireo, a novel framework that combines depth and language cues to improve open-vocabulary, domain-generalized semantic segmentation, achieving state-of-the-art results in diverse environments.

Contribution

Vireo is the first unified single-stage framework that leverages depth and language for open-vocabulary domain-generalized segmentation, enhancing robustness and scalability.

Findings

Vireo outperforms existing methods in domain generalization and open-vocabulary recognition.

The proposed components improve convergence speed and textual influence.

Vireo achieves state-of-the-art performance on benchmark datasets.

Abstract

Open-Vocabulary semantic segmentation (OVSS) and domain generalization in semantic segmentation (DGSS) highlight a subtle complementarity that motivates Open-Vocabulary Domain-Generalized Semantic Segmentation (OV-DGSS). OV-DGSS aims to generate pixel-level masks for unseen categories while maintaining robustness across unseen domains, a critical capability for real-world scenarios such as autonomous driving in adverse conditions. We introduce Vireo, a novel single-stage framework for OV-DGSS that unifies the strengths of OVSS and DGSS for the first time. Vireo builds upon the frozen Visual Foundation Models (VFMs) and incorporates scene geometry via Depth VFMs to extract domain-invariant structural features. To bridge the gap between visual and textual modalities under domain shift, we propose three key components: (1) GeoText Prompts, which align geometric features with language cues and progressively refine VFM encoder representations; (2) Coarse Mask Prior Embedding (CMPE) for enhancing gradient flow for faster convergence and stronger textual influence; and (3) the Domain-Open-Vocabulary Vector Embedding Head (DOV-VEH), which fuses refined structural and semantic features for robust prediction. Comprehensive evaluation on these components demonstrates the effectiveness of our designs. Our proposed Vireo achieves the state-of-the-art performance and surpasses existing methods by a large margin in both domain generalization and open-vocabulary recognition, offering a unified and scalable solution for robust visual understanding in diverse and dynamic environments. Code is available at https://github.com/anonymouse-9c53tp182bvz/Vireo.