Rethinking Query-based Transformer for Continual Image Segmentation

TL;DR

This paper introduces SimCIS, a simple and effective baseline for continual image segmentation that improves plasticity and reduces catastrophic forgetting by directly aligning image features with queries and employing a novel replay mechanism.

Contribution

It proposes SimCIS, a new method that enhances query-based transformers for continual segmentation by direct feature-query alignment and cross-stage consistency, addressing key issues in existing decoupled frameworks.

Findings

SimCIS outperforms state-of-the-art methods across various tasks and settings.

Direct feature-query alignment preserves objectness and improves plasticity.

Cross-stage consistency and visual query replay reduce catastrophic forgetting.

Abstract

Class-incremental/Continual image segmentation (CIS) aims to train an image segmenter in stages, where the set of available categories differs at each stage. To leverage the built-in objectness of query-based transformers, which mitigates catastrophic forgetting of mask proposals, current methods often decouple mask generation from the continual learning process. This study, however, identifies two key issues with decoupled frameworks: loss of plasticity and heavy reliance on input data order. To address these, we conduct an in-depth investigation of the built-in objectness and find that highly aggregated image features provide a shortcut for queries to generate masks through simple feature alignment. Based on this, we propose SimCIS, a simple yet powerful baseline for CIS. Its core idea is to directly select image features for query assignment, ensuring "perfect alignment" to preserve objectness, while simultaneously allowing queries to select new classes to promote plasticity. To further combat catastrophic forgetting of categories, we introduce cross-stage consistency in selection and an innovative "visual query"-based replay mechanism. Experiments demonstrate that SimCIS consistently outperforms state-of-the-art methods across various segmentation tasks, settings, splits, and input data orders. All models and codes will be made publicly available at https://github.com/SooLab/SimCIS.