StylePrompter: Enhancing Domain Generalization with Test-Time Style Priors

TL;DR

StylePrompter introduces a novel test-time style prompt mechanism in the language modality to dynamically adapt trained models for improved domain generalization to unseen domains, leveraging style embeddings and regularization.

Contribution

The paper proposes a new style prompt approach that uses language-based style priors and a style prompter to enhance domain generalization in vision models.

Findings

Achieves state-of-the-art results on multiple datasets.

Effectively handles data from unknown domains.

Demonstrates robustness through style regularization.

Abstract

In real-world applications, the sample distribution at the inference stage often differs from the one at the training stage, causing performance degradation of trained deep models. The research on domain generalization (DG) aims to develop robust algorithms that can improve the generalized performance in unseen domains by training on a few domains. However, the domain-agnostic vision model, trained on a limited number of domains using traditional domain generalization methods, cannot guarantee its effectiveness in dealing with unseen domains. The introduction of language can break the closed cognition space of the vision model, providing additional semantic information that cannot be inferred from vision-only datasets. In this paper, we propose to overcome the challenge in previous DG methods by introducing the style prompt in the language modality to adapt the trained model dynamically. In particular, we train a style prompter to extract style information of the current image into an embedding in the token embedding space and place it in front of the candidate category words as prior knowledge to prompt the model. Our open space partition of the style token embedding space and the hand-crafted style regularization enable the trained style prompter to handle data from unknown domains effectively. Extensive experiments verify the effectiveness of our method and demonstrate state-of-the-art performances on multiple public datasets. Codes will be available after the acceptance of this paper.