Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weighting

TL;DR

This paper introduces Dynamic Prefix Weighting (DPW), a novel method for improving continual learning in vision-language models by dynamically adjusting prefix weights and selectively utilizing adapters, leading to state-of-the-art results.

Contribution

The paper proposes DPW, a framework that dynamically weights prefixes and adapters based on input importance, enhancing continual learning in vision-language models.

Findings

Achieves state-of-the-art performance in domain-class incremental learning for VLMs.

Demonstrates effective dynamic adjustment of prefix and adapter weights.

Improves model adaptation to downstream tasks.

Abstract

We investigate recently introduced domain-class incremental learning scenarios for vision-language models (VLMs). Recent works address this challenge using parameter-efficient methods, such as prefix-tuning or adapters, which facilitate model adaptation to downstream tasks by incorporating task-specific information into input tokens through additive vectors. However, previous approaches often normalize the weights of these vectors, disregarding the fact that different input tokens require different degrees of adjustment. To overcome this issue, we propose Dynamic Prefix Weighting (DPW), a framework that dynamically assigns weights to prefixes, complemented by adapters. DPW consists of 1) a gating module that adjusts the weights of each prefix based on the importance of the corresponding input token, and 2) a weighting mechanism that derives adapter output weights as a residual of prefix-tuning weights, ensuring that adapters are utilized only when necessary. Experimental results demonstrate that our method achieves state-of-the-art performance in domain-class incremental learning scenarios for VLMs. The code is available at: https://github.com/YonseiML/dpw.