GA2-CLIP: Generic Attribute Anchor for Efficient Prompt Tuningin Video-Language Models

TL;DR

GA2-CLIP introduces a novel prompt tuning framework for video-language models that enhances generalization by using external prompts and attribute anchors, outperforming existing methods.

Contribution

The paper proposes a plug-and-play coupling prompt learning method with attribute anchors to improve VLM generalization in video tasks, addressing semantic space narrowing.

Findings

Significantly outperforms state-of-the-art prompt tuning methods.

Improves base-to-new class prediction accuracy.

Effectively maintains attribute relevance and model generalization.

Abstract

Visual and textual soft prompt tuning can effectively improve the adaptability of Vision-Language Models (VLMs) in downstream tasks. However, fine-tuning on video tasks impairs the model's generalization ability to unseen classes. Existing methods attempt to mitigate this forgetting effect by regularizing the gap between hand-crafted prompts and soft prompts, but this also weakens the learning ability of soft prompts. To address this challenge, we propose a plug-and-play coupling prompt learning framework to optimize the generalization performance of V-L models in video tasks, with the core motivation of mitigating semantic space narrowing during fine-tuning by introducing an externally supervised prompt. Specifically, for textual prompts, we introduce pre-trained prompts from other datasets as hard prompt tokens. These are concatenated with soft prompt tokens and coupled via a learnable mapping layer. This competitive prompting approach prevents the semantic space from overfitting to supervised categories. In addition, we introduce a set of well-designed irrelevant video sets and negative prompts as generic attribute anchors to maintain the generic relevance of the attributes in the pre-trained semantic space, thus preserving the generalization ability. Experiments on video tasks demonstrate that our method significantly outperforms state-of-the-art prompt tuning approaches across generalization benchmarks, particularly on base-to-new class prediction.