Dynamic Rank Adaptation for Vision-Language Models

TL;DR

This paper introduces Dynamic Rank Adaptation (DRA), a novel method for fine-tuning vision-language models that dynamically allocates adaptation resources to improve recognition of new, unseen classes while maintaining generalization.

Contribution

DRA adaptively assigns feature ranks based on token importance, enhancing generalization to new classes in vision-language models, a novel approach compared to existing static methods.

Findings

DRA outperforms existing methods on benchmarks for new class recognition.

DRA improves cross-dataset and domain generalization performance.

DRA effectively balances adaptation and preservation of general knowledge.

Abstract

Pre-trained large vision-language models (VLMs) like CLIP demonstrate impressive generalization ability. Existing prompt-based and adapter-based works have made significant progress in fine-tuning VLMs but still face the challenges of maintaining strong generalization abilities, particularly towards unseen new classes. This limitation partly arises from these methods treating all tokens of the image and text encoder equally, which can lead to overfitting on less informative features (e.g., background noise, template words) and degrade the general representations that are crucial for novel concept recognition. To address this issue, we propose Dynamic Rank Adaptation (DRA), a novel adapter variant method, designed specifically to enhance new class generalization. DRA dynamically allocates adaptation ranks based on the importance of features during training to preserve general knowledge. DRA first employs token importance grouping, using sequence attention to evaluate and group tokens by their importance. Then, we adopt rank adaptation according to the importance of each token group dynamically by assigning higher feature ranks to the more important tokens. Also, we design a new channel response mechanism to prioritize the preservation and adaptation of feature channels identified as the most informative for each instance. In addition, a L1 regularization term is introduced to stabilize the training. Extensive experiments demonstrate the effectiveness and superiority of our proposed DRA over existing works, especially on enhancing the performance of new classes on various benchmarks, including base-new classes, cross-datasets evaluation and domain generalization. The source code will be published after the paper is received.