DynAlign: Unsupervised Dynamic Taxonomy Alignment for Cross-Domain Segmentation

TL;DR

DynAlign is an unsupervised framework that leverages foundation models and semantic knowledge to align and adapt to new, fine-grained, or differently named categories in cross-domain semantic segmentation without manual annotations.

Contribution

It introduces a novel unsupervised taxonomy alignment method that integrates foundation models with knowledge fusion to handle label-level domain gaps in segmentation tasks.

Findings

Significant performance improvements over existing methods on GTA to Mapillary Vistas and GTA to IDD benchmarks.

Effective in adapting to novel and fine-grained categories without manual annotations.

Demonstrates seamless taxonomy adaptation via model retraining or direct inference.

Abstract

Current unsupervised domain adaptation (UDA) methods for semantic segmentation typically assume identical class labels between the source and target domains. This assumption ignores the label-level domain gap, which is common in real-world scenarios, thus limiting their ability to identify finer-grained or novel categories without requiring extensive manual annotation. A promising direction to address this limitation lies in recent advancements in foundation models, which exhibit strong generalization abilities due to their rich prior knowledge. However, these models often struggle with domain-specific nuances and underrepresented fine-grained categories. To address these challenges, we introduce DynAlign, a framework that integrates UDA with foundation models to bridge both the image-level and label-level domain gaps. Our approach leverages prior semantic knowledge to align source categories with target categories that can be novel, more fine-grained, or named differently (e.g., vehicle to {car, truck, bus}). Foundation models are then employed for precise segmentation and category reassignment. To further enhance accuracy, we propose a knowledge fusion approach that dynamically adapts to varying scene contexts. DynAlign generates accurate predictions in a new target label space without requiring any manual annotations, allowing seamless adaptation to new taxonomies through either model retraining or direct inference. Experiments on the street scene semantic segmentation benchmarks GTA to Mapillary Vistas and GTA to IDD validate the effectiveness of our approach, achieving a significant improvement over existing methods. Our code will be publicly available.