Fine-tuning Pre-trained Vision-Language Models in a Human-Annotation-Free Manner

TL;DR

This paper introduces CoFT, an unsupervised framework for adapting large vision-language models to downstream tasks without labeled data, using dual-model collaboration and prompt strategies to improve robustness and performance.

Contribution

It presents a novel unsupervised adaptation method, CoFT, that leverages dual-model collaboration and prompt-based pseudo-label filtering, eliminating the need for manual thresholds.

Findings

CoFT outperforms existing unsupervised methods in various tasks.

CoFT+ further improves performance through iterative and contrastive learning.

The approach achieves results comparable to few-shot supervised baselines.

Abstract

Large-scale vision-language models (VLMs) such as CLIP exhibit strong zero-shot generalization, but adapting them to downstream tasks typically requires costly labeled data. Existing unsupervised self-training methods rely on pseudo-labeling, yet often suffer from unreliable confidence filtering, confirmation bias, and underutilization of low-confidence samples. We propose Collaborative Fine-Tuning (CoFT), an unsupervised adaptation framework that leverages unlabeled data through a dual-model, cross-modal collaboration mechanism. CoFT introduces a dual-prompt learning strategy with positive and negative textual prompts to explicitly model pseudo-label cleanliness in a sample-dependent manner, removing the need for hand-crafted thresholds or noise assumptions. The negative prompt also regularizes lightweight visual adaptation modules, improving robustness under noisy supervision. CoFT employs a two-phase training scheme, transitioning from parameter-efficient fine-tuning on high-confidence samples to full fine-tuning guided by collaboratively filtered pseudo-labels. Building on CoFT, CoFT+ further enhances adaptation via iterative fine-tuning, momentum contrastive learning, and LLM-generated prompts. Extensive experiments demonstrate consistent gains over existing unsupervised methods and even few-shot supervised baselines.