CluMo: Cluster-based Modality Fusion Prompt for Continual Learning in Visual Question Answering

TL;DR

CluMo introduces a novel prompt-based continual learning method for vision-language models, using cluster-based modality fusion prompts to improve generalization and mitigate catastrophic forgetting in visual question answering tasks.

Contribution

The paper proposes CluMo, a new prompt-based continual learning approach with a cluster-based modality fusion prompt and a two-stage training strategy for VLMs.

Findings

Achieves state-of-the-art performance on two benchmarks.

Effectively mitigates catastrophic forgetting in continual learning.

Enhances generalization in multimodal VQA tasks.

Abstract

Large vision-language models (VLMs) have shown significant performance boost in various application domains. However, adopting them to deal with several sequentially encountered tasks has been challenging because finetuning a VLM on a task normally leads to reducing its generalization power and the capacity of learning new tasks as well as causing catastrophic forgetting on previously learned tasks. Enabling using VLMs in multimodal continual learning (CL) settings can help to address such scenarios. To improve generalization capacity and prevent catastrophic forgetting, we propose a novel prompt-based CL method for VLMs, namely $\textbf{Clu}$ster-based $\textbf{Mo}$dality Fusion Prompt (\textbf{CluMo}). We design a novel \textbf{Key-Key-Prompt} pair, where each prompt is associated with a visual prompt key and a textual prompt key. We adopt a two-stage training strategy. During the first stage, the single-modal keys are trained via $K$-means clustering algorithm to help select the best semantically matched prompt. During the second stage, the prompt keys are frozen, the selected prompt is attached to the input for training the VLM in the CL scenario. Experiments on two benchmarks demonstrate that our method achieves SOTA performance.