PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning

TL;DR

PAL introduces a novel exemplar-free framework for multi-modal class-incremental learning that effectively handles missing modalities by using modality-specific prompts and an analytical solution, reducing forgetting and improving performance.

Contribution

The paper proposes PAL, a new method that addresses missing modality issues in MMCIL using prompts and an analytical approach, which was not explored in prior work.

Findings

PAL outperforms existing methods on UPMC-Food101 and N24News datasets.

It effectively mitigates forgetting in multi-modal incremental learning.

PAL demonstrates robustness to missing modalities across various scenarios.

Abstract

Multi-modal class-incremental learning (MMCIL) seeks to leverage multi-modal data, such as audio-visual and image-text pairs, thereby enabling models to learn continuously across a sequence of tasks while mitigating forgetting. While existing studies primarily focus on the integration and utilization of multi-modal information for MMCIL, a critical challenge remains: the issue of missing modalities during incremental learning phases. This oversight can exacerbate severe forgetting and significantly impair model performance. To bridge this gap, we propose PAL, a novel exemplar-free framework tailored to MMCIL under missing-modality scenarios. Concretely, we devise modality-specific prompts to compensate for missing information, facilitating the model to maintain a holistic representation of the data. On this foundation, we reformulate the MMCIL problem into a Recursive Least-Squares task, delivering an analytical linear solution. Building upon these, PAL not only alleviates the inherent under-fitting limitation in analytic learning but also preserves the holistic representation of missing-modality data, achieving superior performance with less forgetting across various multi-modal incremental scenarios. Extensive experiments demonstrate that PAL significantly outperforms competitive methods across various datasets, including UPMC-Food101 and N24News, showcasing its robustness towards modality absence and its anti-forgetting ability to maintain high incremental accuracy.