CRoF: CLIP-based Robust Few-shot Learning on Noisy Labels

TL;DR

This paper introduces CRoF, a plug-in module for CLIP that improves few-shot learning robustness on noisy labels by using discriminative prompts and a weighted fine-tuning strategy, outperforming existing methods.

Contribution

CRoF is a novel plug-in that enhances CLIP's domain generalization on noisy data through task-oriented prompts and a weighted multi-label loss.

Findings

CRoF outperforms fine-tuned and vanilla CLIP on various noise types.

Discriminative prompts increase inter-class textual embedding distances.

Weighted multi-label loss improves robustness under noisy labels.

Abstract

Noisy labels threaten the robustness of few-shot learning (FSL) due to the inexact features in a new domain. CLIP, a large-scale vision-language model, performs well in FSL on image-text embedding similarities, but it is susceptible to misclassification caused by noisy labels. How to enhance domain generalization of CLIP on noisy data within FSL tasks is a critical challenge. In this paper, we provide a novel view to mitigate the influence of noisy labels, CLIP-based Robust Few-shot learning (CRoF). CRoF is a general plug-in module for CLIP-based models. To avoid misclassification and confused label embedding, we design the few-shot task-oriented prompt generator to give more discriminative descriptions of each category. The proposed prompt achieves larger distances of inter-class textual embedding. Furthermore, rather than fully trusting zero-shot classification by CLIP, we fine-tune CLIP on noisy few-shot data in a new domain with a weighting strategy like label-smooth. The weights for multiple potentially correct labels consider the relationship between CLIP's prior knowledge and original label information to ensure reliability. Our multiple label loss function further supports robust training under this paradigm. Comprehensive experiments show that CRoF, as a plug-in, outperforms fine-tuned and vanilla CLIP models on different noise types and noise ratios.