Robust Prompt Tuning for Vision-Language Models with Mild Semantic Noise

TL;DR

This paper introduces ANPrompt, a novel prompt tuning framework for vision-language models that actively incorporates weak semantic noise and stabilizes visual semantics, significantly improving robustness and generalization to unseen categories.

Contribution

The paper proposes ANPrompt, which integrates weak semantic noise and a noise-resistant visual prompt, along with a new loss function, to enhance robustness and generalization in prompt tuning.

Findings

Outperforms existing methods on 11 benchmarks.

Enhances robustness to semantic noise.

Improves generalization to unseen categories.

Abstract

Prompt tuning has shown promising results, but its robustness and generalization to unseen categories remain limited. Through our experiments, we demonstrate that the complete removal of semantic noise is a key factor restricting robustness. Existing methods typically suppress or filter out semantic noise in the prompt space, inadvertently hindering the model's robustness and its ability to generalize to unseen categories. To address this, we propose ANPrompt, a robust prompt tuning framework that actively incorporates weak semantic noise. By clustering weakly perturbed features into noise prompts and integrating them with learnable tokens in both the text and vision encoders, ANPrompt ensures controlled exposure to semantic variations. To enhance the visual pathway, we introduce the Noise-Resistant Visual Prompt Prototype (NRVPP), which stabilizes visual semantics under weak perturbations. Additionally, we propose a Weak Alignment Loss (WALoss) at the logits level to enforce consistency between clean and perturbed predictions, providing stable supervision. By combining weak semantic noise exposure with logits-based consistency, ANPrompt prevents overfitting to specific phrasings while preserving semantic integrity. Extensive experiments across 11 benchmarks, including base-to-new splits, show that ANPrompt consistently outperforms existing prompt tuning methods, offering superior robustness to semantic noise and improved generalization across tasks.