The Finer the Better: Towards Granular-aware Open-set Domain Generalization

TL;DR

This paper introduces SeeCLIP, a semantic-enhanced framework for open-set domain generalization that improves fine-grained discrimination and unknown detection by leveraging semantic tokens, contrastive learning, and pseudo-unknown synthesis.

Contribution

The paper proposes a novel semantic-aware prompt enhancement, duplex contrastive learning, and semantic-guided diffusion to improve open-set domain generalization with fine-grained semantic understanding.

Findings

Achieves 3% higher accuracy over state-of-the-art methods.

Attains 5% higher H-score in benchmarks.

Demonstrates effectiveness across five diverse datasets.

Abstract

Open-Set Domain Generalization (OSDG) tackles the realistic scenario where deployed models encounter both domain shifts and novel object categories. Despite impressive progress with vision-language models like CLIP, existing methods still fall into the dilemma between structural risk of known-classes and open-space risk from unknown-classes, and easily suffers from over-confidence, especially when distinguishing ``hard unknowns" that share fine-grained visual similarities with known classes. To this end, we propose a Semantic-enhanced CLIP (SeeCLIP) framework that explicitly addresses this dilemma through fine-grained semantic enhancement. In SeeCLIP, we propose a semantic-aware prompt enhancement module to decompose images into discriminative semantic tokens, enabling nuanced vision-language alignment beyond coarse category labels. To position unknown prompts effectively, we introduce duplex contrastive learning with complementary objectives, that is, repulsion to maintain separability from known classes, and cohesion to preserve semantic proximity. Further, our semantic-guided diffusion module synthesizes pseudo-unknowns by perturbing extracted semantic tokens, generating challenging samples that are visually similar to known classes yet exhibit key local differences. These hard negatives force the model to learn finer decision boundaries. Extensive experiments across five benchmarks demonstrate consistent improvements of 3% accuracy and 5% H-score over state-of-the-art methods.