Debiased Fine-Tuning for Vision-language Models by Prompt Regularization

TL;DR

This paper introduces Prompt Regularization (ProReg), a fine-tuning method for vision-language models that leverages prompt-based predictions to prevent overfitting and improve out-of-distribution performance.

Contribution

ProReg is a novel fine-tuning approach that uses prompt-based regularization with adaptive weighting to better utilize pretraining knowledge and reduce bias from downstream data.

Findings

ProReg outperforms traditional fine-tuning and prompt tuning on various benchmarks.

It effectively reduces overfitting by leveraging prompt-based predictions.

ProReg demonstrates strong out-of-distribution generalization.

Abstract

We present a new paradigm for fine-tuning large-scale visionlanguage pre-trained models on downstream task, dubbed Prompt Regularization (ProReg). Different from traditional fine-tuning which easily overfits to the downstream task data, ProReg uses the prediction by prompting the pretrained model to regularize the fine-tuning. The motivation is: by prompting the large model "a photo of a [CLASS]", the fil-lin answer is only dependent on the pretraining encyclopedic knowledge while independent of the task data distribution, which is usually biased. Specifically, given a training sample prediction during fine-tuning, we first calculate its KullbackLeibler loss of the prompt prediction and Cross-Entropy loss of the ground-truth label, and then combine them with a proposed sample-wise adaptive trade-off weight, which automatically adjusts the transfer between the pretrained and downstream domains. On various out-of-distribution benchmarks, we show the consistently strong performance of ProReg compared with conventional fine-tuning, zero-shot prompt, prompt tuning, and other state-of-the-art methods.