Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning

TL;DR

This paper introduces D-MoLE, a novel architecture-evolving method for continual multimodal instruction tuning that dynamically allocates model capacity to improve adaptation to new tasks while preserving prior knowledge.

Contribution

It proposes a dynamic, architecture-evolving approach with expert allocation and curriculum learning to address task conflicts and modality imbalance in continual MLLM training.

Findings

Achieves 15% average improvement over baselines.

First study of architectural continual learning for MLLMs.

Effectively balances modality updates and task adaptation.

Abstract

Continual multimodal instruction tuning is crucial for adapting Multimodal Large Language Models (MLLMs) to evolving tasks. However, most existing methods adopt a fixed architecture, struggling with adapting to new tasks due to static model capacity. We propose to evolve the architecture under parameter budgets for dynamic task adaptation, which remains unexplored and imposes two challenges: 1) task architecture conflict, where different tasks require varying layer-wise adaptations, and 2) modality imbalance, where different tasks rely unevenly on modalities, leading to unbalanced updates. To address these challenges, we propose a novel Dynamic Mixture of Curriculum LoRA Experts (D-MoLE) method, which automatically evolves MLLM's architecture with controlled parameter budgets to continually adapt to new tasks while retaining previously learned knowledge. Specifically, we propose a dynamic layer-wise expert allocator, which automatically allocates LoRA experts across layers to resolve architecture conflicts, and routes instructions layer-wisely to facilitate knowledge sharing among experts. Then, we propose a gradient-based inter-modal continual curriculum, which adjusts the update ratio of each module in MLLM based on the difficulty of each modality within the task to alleviate the modality imbalance problem. Extensive experiments show that D-MoLE significantly outperforms state-of-the-art baselines, achieving a 15% average improvement over the best baseline. To the best of our knowledge, this is the first study of continual learning for MLLMs from an architectural perspective.