A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance

TL;DR

This paper introduces RISE, a novel domain generalization method that distills knowledge from CLIP using text guidance to improve model robustness on unseen domains.

Contribution

It presents a new regularization technique leveraging CLIP's text and image representations for domain generalization, a first in using large vision-language models for this purpose.

Findings

RISE outperforms state-of-the-art methods on benchmark datasets.

Text-guided regularization enhances model generalization.

First application of vision-language knowledge distillation for domain generalization.

Abstract

Domain generalization studies the problem of training a model with samples from several domains (or distributions) and then testing the model with samples from a new, unseen domain. In this paper, we propose a novel approach for domain generalization that leverages recent advances in large vision-language models, specifically a CLIP teacher model, to train a smaller model that generalizes to unseen domains. The key technical contribution is a new type of regularization that requires the student's learned image representations to be close to the teacher's learned text representations obtained from encoding the corresponding text descriptions of images. We introduce two designs of the loss function, absolute and relative distance, which provide specific guidance on how the training process of the student model should be regularized. We evaluate our proposed method, dubbed RISE (Regularized Invariance with Semantic Embeddings), on various benchmark datasets and show that it outperforms several state-of-the-art domain generalization methods. To our knowledge, our work is the first to leverage knowledge distillation using a large vision-language model for domain generalization. By incorporating text-based information, RISE improves the generalization capability of machine learning models.