Dropout Prompt Learning: Towards Robust and Adaptive Vision-Language Models

TL;DR

This paper introduces Dropout Prompt Learning, a novel regularization method applying dropout to tokens in vision-language models to enhance robustness and generalization across various challenging scenarios.

Contribution

It proposes a new dropout technique for tokens in vision-language models with adaptive probabilities and residual entropy regularization for improved robustness.

Findings

Outperforms existing regularization methods in multiple benchmarks.

Enhances low-shot and out-of-distribution generalization.

Achieves significant performance gains in challenging scenarios.

Abstract

Dropout is a widely used regularization technique which improves the generalization ability of a model by randomly dropping neurons. In light of this, we propose Dropout Prompt Learning, which aims for applying dropout to improve the robustness of the vision-language models. Different from the vanilla dropout, we apply dropout on the tokens of the textual and visual branches, where we evaluate the token significance considering both intra-modal context and inter-modal alignment, enabling flexible dropout probabilities for each token. Moreover, to maintain semantic alignment for general knowledge transfer while encouraging the diverse representations that dropout introduces, we further propose residual entropy regularization. Experiments on 15 benchmarks show our method's effectiveness in challenging scenarios like low-shot learning, long-tail classification, and out-of-distribution generalization. Notably, our method surpasses regularization-based methods including KgCoOp by 5.10% and PromptSRC by 2.13% in performance on base-to-novel generalization.