Generalizable Vision-Language Few-Shot Adaptation with Predictive Prompts and Negative Learning

TL;DR

PromptFuseNL is a novel framework that significantly improves few-shot vision-language adaptation by combining predictive prompts, negative learning, and noise-robust strategies, achieving state-of-the-art results efficiently.

Contribution

It introduces a unified approach integrating predictive prompt tuning with negative learning and an unsupervised reweighting strategy for label noise robustness.

Findings

Outperforms existing methods on 15 benchmarks in all shot settings.

Achieves up to 300x faster training and 1000x lower FLOPs.

Sets new state-of-the-art for robust, scalable few-shot vision-language adaptation.

Abstract

Few-shot adaptation remains a core challenge for vision-language models (VLMs), especially under limited supervision and noisy support samples. We propose PromptFuseNL, a unified framework that enhances few-shot generalization by combining predictive prompt tuning with dual-branch positive and negative learning. The method refines class prototypes through task-conditioned residuals, multi-stage cross-modal coordination, and semantic hard negative mining. To address label noise, we introduce an unsupervised instance reweighting strategy that downweights unreliable support examples without requiring additional labels or structural changes. PromptFuseNL fuses visual and textual cues through lightweight modules for efficient and discriminative prediction. Evaluated across 15 benchmarks, it consistently surpasses existing prompt- and adapter-based methods in all shot settings while remaining highly efficient, achieving up to 300x faster training and 1000x lower FLOPs compared to full prompt tuning, achieving a new state-of-the-art for robust and scalable few-shot vision-language adaptation.