AnchorOPT: Towards Optimizing Dynamic Anchors for Adaptive Prompt Learning

TL;DR

AnchorOPT introduces a dynamic, learnable anchor and position matrix for prompt learning with CLIP, enhancing flexibility and performance across tasks without added complexity.

Contribution

It proposes a novel dynamic anchor framework that learns task-specific anchors and adaptively optimizes their position, improving prompt learning flexibility and effectiveness.

Findings

Achieves comparable or superior performance to more complex methods.

Seamlessly integrates into existing frameworks, providing consistent gains.

Operates effectively across diverse datasets.

Abstract

Existing prompt learning methods, which are built upon CLIP models, leverage textual tokens as anchors to guide the learnable soft tokens. This guidance improves CLIP generalizations. However, these anchors-static in both value and position-lack cross-task and stage-adaptive flexibility. To address this limitation, we propose AnchorOPT, a dynamic anchor-based prompt learning framework. Specifically, AnchorOPT introduces dynamism in two key dimensions: (i) anchor values eschew handcrafted explicit textual tokens (e.g., "shape", "color"), instead learning dynamically from task-specific data; and (ii) the positional relationship between anchor and soft tokens is no longer fixed but adaptively optimized via a learnable position matrix conditioned on the training stage and task context. Training occurs in two stages: we first learn the anchor tokens, then freeze and transfer them to the second stage for optimization of soft tokens and the position matrix. Extensive experiments demonstrate that using only a simple learnable anchor and position matrix achieves performance comparable to or exceeding some methods incorporating additional learnable modules or regularization techniques. As a plug-and-play module, AnchorOPT integrates seamlessly into existing frameworks, yielding consistent performance gains across diverse datasets. Code is publicly available at https://github.com/zhengli97/ATPrompt.