StyLIP: Multi-Scale Style-Conditioned Prompt Learning for CLIP-based Domain Generalization

TL;DR

StyLIP is a novel prompt learning approach that improves CLIP's domain generalization by disentangling style and content features, enabling better adaptation to new domains across multiple benchmarks.

Contribution

It introduces style projectors for domain-specific prompt generation and a contrastive training scheme, advancing zero-shot domain generalization in CLIP-based models.

Findings

Outperforms state-of-the-art methods on multiple benchmarks

Effective disentanglement of style and content features

Consistent improvements across five domain generalization settings

Abstract

Large-scale foundation models, such as CLIP, have demonstrated impressive zero-shot generalization performance on downstream tasks, leveraging well-designed language prompts. However, these prompt learning techniques often struggle with domain shift, limiting their generalization capabilities. In our study, we tackle this issue by proposing StyLIP, a novel approach for Domain Generalization (DG) that enhances CLIP's classification performance across domains. Our method focuses on a domain-agnostic prompt learning strategy, aiming to disentangle the visual style and content information embedded in CLIP's pre-trained vision encoder, enabling effortless adaptation to novel domains during inference. To achieve this, we introduce a set of style projectors that directly learn the domain-specific prompt tokens from the extracted multi-scale style features. These generated prompt embeddings are subsequently combined with the multi-scale visual content features learned by a content projector. The projectors are trained in a contrastive manner, utilizing CLIP's fixed vision and text backbones. Through extensive experiments conducted in five different DG settings on multiple benchmark datasets, we consistently demonstrate that StyLIP outperforms the current state-of-the-art (SOTA) methods.