Neutral-Reference Prompting for Vision-Language Models

TL;DR

This paper introduces NeRP, a prompting correction method that enhances vision-language models' ability to recognize unseen classes without sacrificing accuracy on known classes, by leveraging neutral prompts and reference images.

Contribution

NeRP is a novel, plug-and-play prompting correction strategy that improves unseen class recognition in VLMs without changing model parameters.

Findings

NeRP significantly boosts accuracy on unseen classes across multiple benchmarks.

NeRP maintains high performance on known classes while improving generalization.

Extensive experiments validate NeRP's effectiveness across various backbones and data settings.

Abstract

Efficient transfer learning of vision-language models (VLMs) commonly suffers from a Base-New Trade-off (BNT): improving performance on unseen (new) classes often degrades accuracy on known (base) classes. Addressing how to boost recognition of unseen classes without sacrificing known-class performance remains a central challenge. Existing work often simplistically attributes the BNT to overfitting on known classes. We observe an interesting phenomenon: VLMs frequently exhibit asymmetric confusion on certain downstream data, i.e., samples of class A are systematically mispredicted as class B, while the reverse confusion (B to A) rarely occurs. For known classes, this kind of bias can be mitigated by tuning using a cross-entropy loss, but for unseen classes, such pretraining-induced bias persists and harms generalization. Motivated by this, we propose NeRP, a plug-and-play prompting correction strategy that improves discrimination on unseen classes without modifying model parameters. NeRP leverages neutral text prompts and reference images to measure class-wise prior preferences along the pre-trained inter-class geometry, and combines them with the sample likelihood to obtain the model's surrogate score. If, for a given sample, the prior strongly favors the current prediction while the observed evidence is clearly insufficient, we perform a local flip between easily confusable class pairs, thereby correcting prior-dominated mispredictions. Extensive experiments across multiple backbones and 15 few-shot and cross-domain benchmarks show that NeRP substantially improves accuracy on unseen classes while preserving known-class prediction performance.