Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework

TL;DR

This paper introduces Continual-NExT, a framework for lifelong learning in Dual-to-Dual Multimodal Large Language Models, addressing challenges like catastrophic forgetting and knowledge transfer with a novel MAGE method.

Contribution

The paper proposes a standardized continual learning framework and a new MAGE method to enhance knowledge transfer and reduce forgetting in Dual-to-Dual MLLMs.

Findings

MAGE outperforms existing continual learning methods.

Continual-NExT achieves state-of-the-art results.

Framework effectively mitigates catastrophic forgetting.

Abstract

Dual-to-Dual MLLMs refer to Multimodal Large Language Models, which can enable unified multimodal comprehension and generation through text and image modalities. Although exhibiting strong instantaneous learning and generalization capabilities, Dual-to-Dual MLLMs still remain deficient in lifelong evolution, significantly affecting continual adaptation to dynamic real-world scenarios. One of the challenges is that learning new tasks inevitably destroys the learned knowledge. Beyond traditional catastrophic forgetting, Dual-to-Dual MLLMs face other challenges, including hallucination, instruction unfollowing, and failures in cross-modal knowledge transfer. However, no standardized continual learning framework for Dual-to-Dual MLLMs has been established yet, leaving these challenges unexplored. Thus, in this paper, we establish Continual-NExT, a continual learning framework for Dual-to-Dual MLLMs with deliberately-architected evaluation metrics. To improve the continual learning capability of Dual-to-Dual MLLMs, we propose an efficient MAGE (Mixture and Aggregation of General LoRA and Expert LoRA) method to further facilitate knowledge transfer across modalities and mitigate forgetting. Extensive experiments demonstrate that MAGE outperforms other continual learning methods and achieves state-of-the-art performance.