SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

TL;DR

SAME introduces a stabilized mixture-of-experts approach for multimodal continual instruction tuning, addressing expert and router drift to improve task retention and model performance in evolving data environments.

Contribution

It proposes a novel stabilization method for expert routing and update regulation, enhancing continual learning in multimodal models without rehearsal.

Findings

Achieves state-of-the-art results on multimodal continual instruction benchmarks.

Effectively mitigates expert and router drift in dynamic data scenarios.

Reduces redundant computation through adaptive expert activation.

Abstract

Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually expand their capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential. Recent methods leverage sparse expert routing to promote task specialization, but we find that the expert routing process suffers from drift as the data distribution evolves. For example, a grounding query that previously activated localization experts may instead be routed to irrelevant experts after learning OCR tasks. Meanwhile, the grounding-related experts can be overwritten by new tasks and lose their original functionality. Such failure reflects two problems: router drift, where expert selection becomes inconsistent over time, and expert drift, where shared experts are overwritten across tasks. Therefore, we propose StAbilized Mixture-of-Experts (SAME) for MCIT. To address router drift, SAME stabilizes expert selection by decomposing routing dynamics into orthogonal subspaces and updating only task-relevant directions. To mitigate expert drift, we regulate expert updates via curvature-aware scaling using historical input covariance in a rehearsal-free manner. SAME also introduces adaptive expert activation to freeze selected experts during training, reducing redundant computation and cross-task interference. Extensive experiments demonstrate its SOTA performance.