CRISP: Contrastive Residual Injection and Semantic Prompting for Continual Video Instance Segmentation

TL;DR

CRISP introduces a novel framework combining contrastive residual injection and semantic prompting to improve continual video instance segmentation, effectively balancing learning new categories while retaining previous knowledge.

Contribution

It proposes a new method with instance-wise, category-wise, and task-wise modules, addressing confusion and catastrophic forgetting in continual segmentation tasks.

Findings

Outperforms existing methods on YouTube-VIS datasets

Effectively reduces catastrophic forgetting

Improves segmentation and classification accuracy

Abstract

Continual video instance segmentation demands both the plasticity to absorb new object categories and the stability to retain previously learned ones, all while preserving temporal consistency across frames. In this work, we introduce Contrastive Residual Injection and Semantic Prompting (CRISP), an earlier attempt tailored to address the instance-wise, category-wise, and task-wise confusion in continual video instance segmentation. For instance-wise learning, we model instance tracking and construct instance correlation loss, which emphasizes the correlation with the prior query space while strengthening the specificity of the current task query. For category-wise learning, we build an adaptive residual semantic prompt (ARSP) learning framework, which constructs a learnable semantic residual prompt pool generated by category text and uses an adjustive query-prompt matching mechanism to build a mapping relationship between the query of the current task and the semantic residual prompt. Meanwhile, a semantic consistency loss based on the contrastive learning is introduced to maintain semantic coherence between object queries and residual prompts during incremental training. For task-wise learning, to ensure the correlation at the inter-task level within the query space, we introduce a concise yet powerful initialization strategy for incremental prompts. Extensive experiments on YouTube-VIS-2019 and YouTube-VIS-2021 datasets demonstrate that CRISP significantly outperforms existing continual segmentation methods in the long-term continual video instance segmentation task, avoiding catastrophic forgetting and effectively improving segmentation and classification performance. The code is available at https://github.com/01upup10/CRISP.