Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalization

TL;DR

This paper presents SCMD, a novel method leveraging CLIP for domain generalization by selectively distilling hard-to-learn samples, resulting in improved robustness across unseen domains.

Contribution

It introduces a unique sample selection framework and a cross-modality module that enhances domain generalization using large vision-language models.

Findings

SCMD achieves state-of-the-art performance on multiple benchmarks.

The selection strategy effectively identifies hard-to-learn samples.

Theoretical analysis supports the effectiveness of the selection method.

Abstract

Domain Generalization (DG), a crucial research area, seeks to train models across multiple domains and test them on unseen ones. In this paper, we introduce a novel approach, namely, Selective Cross-Modality Distillation for Domain Generalization (SCMD). SCMD leverages the capabilities of large vision-language models, specifically CLIP, to train a more efficient model, ensuring it acquires robust generalization capabilities across unseen domains. Our primary contribution is a unique selection framework strategically designed to identify hard-to-learn samples for distillation. In parallel, we introduce a novel cross-modality module that seamlessly combines the projected features of the student model with the text embeddings from CLIP, ensuring the alignment of similarity distributions. We assess SCMD's performance on various benchmarks, where it empowers a ResNet50 to deliver state-of-the-art performance, surpassing existing domain generalization methods. Furthermore, we provide a theoretical analysis of our selection strategy, offering deeper insight into its effectiveness and potential in the field of DG.