Decoupling Continual Semantic Segmentation

TL;DR

DecoupleCSS introduces a two-stage framework for continual semantic segmentation that decouples class detection from segmentation, improving knowledge retention and adaptability in dense prediction tasks.

Contribution

It proposes a novel two-stage decoupling approach using pre-trained encoders and SAM, enhancing continual learning in semantic segmentation.

Findings

Achieves state-of-the-art performance on multiple CSS benchmarks.

Effectively balances knowledge retention and learning new classes.

Demonstrates robustness across diverse challenging tasks.

Abstract

Continual Semantic Segmentation (CSS) requires learning new classes without forgetting previously acquired knowledge, addressing the fundamental challenge of catastrophic forgetting in dense prediction tasks. However, existing CSS methods typically employ single-stage encoder-decoder architectures where segmentation masks and class labels are tightly coupled, leading to interference between old and new class learning and suboptimal retention-plasticity balance. We introduce DecoupleCSS, a novel two-stage framework for CSS. By decoupling class-aware detection from class-agnostic segmentation, DecoupleCSS enables more effective continual learning, preserving past knowledge while learning new classes. The first stage leverages pre-trained text and image encoders, adapted using LoRA, to encode class-specific information and generate location-aware prompts. In the second stage, the Segment Anything Model (SAM) is employed to produce precise segmentation masks, ensuring that segmentation knowledge is shared across both new and previous classes. This approach improves the balance between retention and adaptability in CSS, achieving state-of-the-art performance across a variety of challenging tasks. Our code is publicly available at: https://github.com/euyis1019/Decoupling-Continual-Semantic-Segmentation.