Domain Prompt Learning for Efficiently Adapting CLIP to Unseen Domains

TL;DR

This paper introduces Domain Prompt Learning (DPL), a lightweight method to adapt CLIP for unseen domain generalization in image classification, significantly improving zero-shot accuracy across multiple benchmarks.

Contribution

The paper proposes DPL, a novel prompt generation approach, enabling effective domain adaptation of CLIP without extensive fine-tuning, enhancing zero-shot performance in DG tasks.

Findings

DPL improves zero-shot CLIP accuracy from 73.7% to 79.3%.

DPL achieves significant gains on PACS, VLCS, OfficeHome, TerraIncognita.

A lightweight prompt generator suffices for effective domain adaptation.

Abstract

Domain generalization (DG) is a difficult transfer learning problem aiming to learn a generalizable model for unseen domains. Recent foundation models (FMs) are robust to many distribution shifts and, therefore, should substantially improve the performance of DG. In this work, we study generic ways to adopt CLIP, a Visual-Language Foundation Model, for DG problems in image classification. While ERM greatly improves the accuracy with bigger backbones and training datasets using standard DG benchmarks, fine-tuning FMs is not practical in many real-world situations. We propose Domain Prompt Learning (DPL) as a novel approach for domain inference in the form of conditional prompt generation. DPL achieved a significant accuracy improvement with only training a lightweight prompt generator (a three-layer MLP), whose parameter is of equivalent scale to the classification projector in the previous DG literature. Combining \dplshort~with CLIP provides surprising performance, raising the accuracy of zero-shot CLIP from 73.7% to 79.3% on several standard datasets, namely PACS, VLCS, OfficeHome, and TerraIncognita. We hope the simplicity and success of our approach lead to broader adoption and analysis of foundation models in the domain generalization field. Our code is available at https://github.com/shogi880/DPLCLIP.