Evolving Without Ending: Unifying Multimodal Incremental Learning for Continual Panoptic Perception

TL;DR

This paper introduces a unified continual learning framework for multimodal and multi-task panoptic perception, addressing catastrophic forgetting and semantic obfuscation to improve comprehensive image understanding in incremental scenarios.

Contribution

It proposes an end-to-end CPP model with a collaborative cross-modal encoder, contrastive knowledge distillation, and a cross-modal consistency constraint, advancing multimodal continual learning.

Findings

Outperforms existing methods on multimodal datasets.

Effectively mitigates catastrophic forgetting in multi-task CL.

Enhances semantic alignment across modalities during incremental learning.

Abstract

Continual learning (CL) is a great endeavour in developing intelligent perception AI systems. However, the pioneer research has predominantly focus on single-task CL, which restricts the potential in multi-task and multimodal scenarios. Beyond the well-known issue of catastrophic forgetting, the multi-task CL also brings semantic obfuscation across multimodal alignment, leading to severe model degradation during incremental training steps. In this paper, we extend CL to continual panoptic perception (CPP), integrating multimodal and multi-task CL to enhance comprehensive image perception through pixel-level, instance-level, and image-level joint interpretation. We formalize the CL task in multimodal scenarios and propose an end-to-end continual panoptic perception model. Concretely, CPP model features a collaborative cross-modal encoder (CCE) for multimodal embedding. We also propose a malleable knowledge inheritance module via contrastive feature distillation and instance distillation, addressing catastrophic forgetting from task-interactive boosting manner. Furthermore, we propose a cross-modal consistency constraint and develop CPP+, ensuring multimodal semantic alignment for model updating under multi-task incremental scenarios. Additionally, our proposed model incorporates an asymmetric pseudo-labeling manner, enabling model evolving without exemplar replay. Extensive experiments on multimodal datasets and diverse CL tasks demonstrate the superiority of the proposed model, particularly in fine-grained CL tasks.