Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models

TL;DR

This paper addresses catastrophic forgetting in unified multimodal generative models by identifying inter-modal forgetting, providing a theoretical explanation, and proposing a novel architecture called MoDE that decouples modalities to improve continual learning performance.

Contribution

The paper introduces MoDE, a lightweight architecture that decouples modalities in UMGMs to mitigate both intra- and inter-modal forgetting during continual learning.

Findings

MoDE significantly reduces inter- and intra-modal forgetting.

MoDE outperforms prior continual learning baselines in multimodal tasks.

Theoretical analysis explains gradient conflicts causing inter-modal forgetting.

Abstract

Unified Multimodal Generative Models (UMGMs) unify visual understanding and image generation within a single autoregressive framework. However, their ability to continually learn new tasks is severely hindered by catastrophic forgetting, both within a modality (intra-modal) and across modalities (inter-modal). While intra-modal forgetting has been studied in prior continual learning (CL) work, inter-modal forgetting remains largely unexplored. In this paper, we identify and empirically validate this phenomenon in UMGMs and provide a theoretical explanation rooted in gradient conflict between modalities. To address both intra- and inter-modal forgetting, we propose Modality-Decoupled Experts (MoDE), a lightweight and scalable architecture that isolates modality-specific updates to mitigate the gradient conflict and leverages knowledge distillation to prevent catastrophic forgetting and preserve pre-trained capabilities. Unlike previous CL methods that remain modality-coupled and suffer from modality gradient conflict, MoDE explicitly decouples modalities to prevent interference. Experiments across diverse benchmarks demonstrate that MoDE significantly mitigates both inter- and intra-modal forgetting, outperforming prior CL baselines in unified multimodal generation settings. Codes will be publicly available: https://github.com/Christina200/MoDE-official.git