Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversification

TL;DR

This paper introduces a data-free distillation method for vision-language models that uses prompt diversification to synthesize diverse surrogate images, enhancing out-of-distribution generalization without relying on large datasets.

Contribution

It proposes three novel prompt diversification techniques—Mix-Prompt, Random-Prompt, and Contrastive-Prompt—for synthesizing diverse images to improve distribution-agnostic model adaptation.

Findings

Contrastive-Prompt achieves the best out-of-distribution generalization.

The methods effectively synthesize diverse surrogate images.

Improved robustness of student models in OOD scenarios.

Abstract

Data-Free Knowledge Distillation (DFKD) has shown great potential in creating a compact student model while alleviating the dependency on real training data by synthesizing surrogate data. However, prior arts are seldom discussed under distribution shifts, which may be vulnerable in real-world applications. Recent Vision-Language Foundation Models, e.g., CLIP, have demonstrated remarkable performance in zero-shot out-of-distribution generalization, yet consuming heavy computation resources. In this paper, we discuss the extension of DFKD to Vision-Language Foundation Models without access to the billion-level image-text datasets. The objective is to customize a student model for distribution-agnostic downstream tasks with given category concepts, inheriting the out-of-distribution generalization capability from the pre-trained foundation models. In order to avoid generalization degradation, the primary challenge of this task lies in synthesizing diverse surrogate images driven by text prompts. Since not only category concepts but also style information are encoded in text prompts, we propose three novel Prompt Diversification methods to encourage image synthesis with diverse styles, namely Mix-Prompt, Random-Prompt, and Contrastive-Prompt. Experiments on out-of-distribution generalization datasets demonstrate the effectiveness of the proposed methods, with Contrastive-Prompt performing the best.