Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners

TL;DR

This paper introduces CLIP-CITE, a framework for fine-tuning large vision-language models efficiently with minimal overfitting, improving few-shot and cross-domain generalization while maintaining versatility.

Contribution

The paper presents a novel fine-tuning method that refines entire VLMs with minimal parameters, addressing overfitting and catastrophic forgetting in limited supervision scenarios.

Findings

Enhanced performance in few-shot learning tasks.

Improved generalization across domains.

Preserved versatility of VLMs on various datasets.

Abstract

Prompt tuning, which involves training a small set of parameters, effectively enhances the pre-trained Vision-Language Models (VLMs) to downstream tasks. However, they often come at the cost of flexibility and adaptability when the tuned models are applied to different datasets or domains. In this paper, we explore capturing the task-specific information via meticulous refinement of entire VLMs, with minimal parameter adjustments. When fine-tuning the entire VLMs for specific tasks under limited supervision, overfitting and catastrophic forgetting become the defacto factors. To mitigate these issues, we propose a framework named CLIP-CITE via designing a discriminative visual-text task, further aligning the visual-text semantics in a supervision manner, and integrating knowledge distillation techniques to preserve the gained knowledge. Extensive experimental results under few-shot learning, base-to-new generalization, domain generalization, and cross-domain generalization settings, demonstrate that our method effectively enhances the performance on specific tasks under limited supervision while preserving the versatility of the VLMs on other datasets.