Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer

TL;DR

This paper introduces Fwd-Prompt, a prompt-based method that enhances multimodal continual instruction tuning by reducing forgetting and negative transfer, enabling models to adapt efficiently to new tasks without retraining from scratch.

Contribution

The paper proposes Fwd-Prompt, a novel prompt tuning approach that minimizes task interference and reuses pre-trained knowledge, addressing catastrophic forgetting and negative transfer in MCIT.

Findings

Fwd-Prompt achieves state-of-the-art results in MCIT tasks.

It updates fewer parameters and requires no old task samples.

The method effectively reduces negative transfer and forgetting.

Abstract

Multimodal Continual Instruction Tuning (MCIT) enables Multimodal Large Language Models (MLLMs) to meet continuously emerging requirements without expensive retraining. MCIT faces two major obstacles: catastrophic forgetting (where old knowledge is forgotten) and negative forward transfer (where the performance of future tasks is degraded). Although existing methods have greatly alleviated catastrophic forgetting, they still suffer from negative forward transfer. We discover a large discrepancy in different input embeddings by performing singular value decomposition (SVD) on input embeddings. This discrepancy results in the model learning irrelevant information for old and pre-trained tasks, leading to catastrophic forgetting and negative forward transfer. To address these issues, we propose Prompt Tuning with Positive Forward Transfer (Fwd-Prompt), a prompt-based method that projects the prompt gradient to the residual space to minimize interference between tasks and to the pre-trained subspace for reusing pre-trained knowledge. Our experiments demonstrate that Fwd-Prompt achieves state-of-the-art performance while updating fewer parameters and requiring no old samples. Our research illuminates the potential of continuously adapting MLLMs to new tasks under the instruction tuning paradigm and encourages future studies to explore MCIT.