When Adaptation Fails: A Gradient-Based Diagnosis of Collapsed Gating in Vision-Language Prompt Learning

TL;DR

This paper investigates why adaptive gating mechanisms in vision-language prompt learning often fail, revealing causes like gradient imbalance and gate degradation, and clarifying when such adaptation is effective.

Contribution

It provides a systematic diagnostic study identifying failure modes of adaptive gating in prompt learning and offers insights into when adaptive mechanisms are beneficial.

Findings

Adaptive gates often produce constant outputs and negligible gradients.

Two failure modes identified: gradient imbalance and gate degradation.

Clarifies conditions under which adaptive gating is effective or not.

Abstract

Adaptive prompting mechanisms have been proposed to enhance vision-language models by dynamically tailoring prompts to inputs. However, in frozen few-shot prompt learning with CLIP-style backbones, we systematically observe that adaptive gates and prompt-selection modules often collapse: they produce nearly constant outputs, contribute negligible gradient signals, and frequently fail to outperform fixed prompts. To further explore this issue, we present a systematic diagnostic study to uncover the underlying causes and conditions of adaptation failure. Through controlled experiments across datasets and multiple prompt learning architectures, we identify two recurring failure modes: gradient magnitude imbalance and gate degradation. Our findings invite a re-examination of indiscriminately adding architectural complexity in parameter-efficient learning and clarify when prompt-level adaptive gating is, and is not, effective in this regime.