Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Models

TL;DR

Bi-CoG introduces a novel self-training method that enhances vision-language model fine-tuning by leveraging bi-consistency and dynamic pseudo-labeling, improving performance across multiple datasets.

Contribution

The paper presents Bi-CoG, a simple plug-and-play approach that reduces bias and hyperparameter sensitivity in semi-supervised fine-tuning of vision-language models.

Findings

Bi-CoG significantly outperforms existing methods on 14 datasets.

Bi-CoG effectively reduces model bias and hyperparameter sensitivity.

Theoretical analysis supports the robustness of Bi-CoG.

Abstract

Exploiting unlabeled data through semi-supervised learning (SSL) or leveraging pre-trained models via fine-tuning are two prevailing paradigms for addressing label-scarce scenarios. Recently, growing attention has been given to combining fine-tuning of pre-trained vision-language models (VLMs) with SSL, forming the emerging paradigm of semi-supervised fine-tuning. However, existing methods often suffer from model bias and hyperparameter sensitivity, due to reliance on prediction consistency or pre-defined confidence thresholds. To address these limitations, we propose a simple yet effective plug-and-play methodology named $\underline{\textbf{Bi-Co}}$nsistency-$\underline{\textbf{G}}$uided Self-Training (Bi-CoG), which assigns high-quality and low-bias pseudo-labels, by simultaneously exploiting inter-model and intra-model consistency, along with an error-aware dynamic pseudo-label assignment strategy. Both theoretical analysis and extensive experiments over 14 datasets demonstrate the effectiveness of Bi-CoG, which consistently and significantly improves the performance of existing methods.