SHED: Style-Homogenized Embedding Alignment for Domain Generalization

TL;DR

SHED introduces a style-homogenized embedding alignment method for CLIP to improve domain generalization by removing domain-specific styles from embeddings during training and inference.

Contribution

It proposes a novel style-homogenized embedding alignment technique that enhances CLIP's robustness to unseen domains in domain generalization tasks.

Findings

SHED achieves state-of-the-art results on five benchmarks.

Outperforms prior methods significantly, e.g., +4.0% on DomainNet.

Effectively removes domain-specific styles from embeddings.

Abstract

Domain generalization aims to enhance model robustness against unseen domains with embedding distribution shifts. While large-scale vision-language models like CLIP exhibit strong generalization, their direct image-text embedding alignment suffers from inherent information asymmetry: images encode both class semantics and domain-specific styles, whereas text prompts primarily convey basic class cues. This asymmetry hinders generalization to novel domains in realistic scenarios. To address this, we propose Style-Homogenized Embedding alignment for Domain-generalization (SHED), a novel CLIP-based method that aligns style-homogenized embeddings instead of raw representations from encoders in CLIP. During training, SHED removes domain-specific style centroids from both image embeddings computed per source domains and text embeddings which are averaged across diverse prompt templates and stripped of a global centroid. For inference, considering the lack of target domain information, SHED projects diverse textual domain centroids into the visual space and aggregates predictions via membership weighting. Extensive experiments on five benchmarks show SHED achieves state-of-the-art performance, outperforming prior methods significantly (e.g., +4.0\% on DomainNet vs. standard fine-tuning).