Multimodal Continual Learning with MLLMs from Multi-scenario Perspectives

TL;DR

This paper introduces UNIFIER, a continual learning framework for multimodal large language models that effectively adapts to diverse visual scenarios, reducing catastrophic forgetting and improving visual question answering performance.

Contribution

The paper presents UNIFIER, a novel continual learning method with Vision Representation Expansion and Vision Consistency Constraint for multi-scenario visual understanding in MLLMs.

Findings

UNIFIER outperforms state-of-the-art methods in cross-scenario continual learning.

MSVQA dataset covers diverse real-world visual scenarios.

Significant improvements in VQA and F1 scores across multiple scenarios.

Abstract

Multimodal large language models (MLLMs) deployed on devices must adapt to continuously changing visual scenarios such as variations in background and perspective, to effectively perform complex visual tasks. To investigate catastrophic forgetting under real-world scenario shifts, we construct a multimodal visual understanding dataset (MSVQA), covering four distinct scenarios and perspectives: high-altitude, underwater, low-altitude, and indoor environments. Furthermore, we propose UNIFIER (mUltimodal coNtInual learning with MLLMs From multi-scenarIo pERspectives), a continual learning (CL) framework designed to address visual discrepancies while learning different scenarios. Compared to existing CL methods, UNIFIER enables knowledge accumulation within the same scenario and mutual enhancement across different scenarios via Vision Representation Expansion (VRE) and Vision Consistency Constraint (VCC). Experimental results show that UNIFIER improves the last-step VQA scores by 2.70%~10.62% and the last-step F1 scores by 3.40%~7.69% compared to the state-of-the-art method, QUAD, in 20-step cross-scenario continual learning tasks. MSVQA dataset is available at https://huggingface.co/datasets/Kaij00/MSVQA.