Comprehensive Generative Replay for Task-Incremental Segmentation with Concurrent Appearance and Semantic Forgetting

TL;DR

This paper introduces a comprehensive generative replay framework for task-incremental segmentation that synthesizes image-mask pairs to mitigate appearance and semantic forgetting in continual learning scenarios.

Contribution

It proposes a novel Bayesian Joint Diffusion model and a Task-Oriented Adapter to improve data synthesis and scalability for diverse segmentation tasks in incremental learning.

Findings

Outperforms existing methods in reducing forgetting in medical image segmentation.

Effective in maintaining performance across cardiac, fundus, and prostate segmentation tasks.

Demonstrates scalability and adaptability to different tasks with high-quality data synthesis.

Abstract

Generalist segmentation models are increasingly favored for diverse tasks involving various objects from different image sources. Task-Incremental Learning (TIL) offers a privacy-preserving training paradigm using tasks arriving sequentially, instead of gathering them due to strict data sharing policies. However, the task evolution can span a wide scope that involves shifts in both image appearance and segmentation semantics with intricate correlation, causing concurrent appearance and semantic forgetting. To solve this issue, we propose a Comprehensive Generative Replay (CGR) framework that restores appearance and semantic knowledge by synthesizing image-mask pairs to mimic past task data, which focuses on two aspects: modeling image-mask correspondence and promoting scalability for diverse tasks. Specifically, we introduce a novel Bayesian Joint Diffusion (BJD) model for high-quality synthesis of image-mask pairs with their correspondence explicitly preserved by conditional denoising. Furthermore, we develop a Task-Oriented Adapter (TOA) that recalibrates prompt embeddings to modulate the diffusion model, making the data synthesis compatible with different tasks. Experiments on incremental tasks (cardiac, fundus and prostate segmentation) show its clear advantage for alleviating concurrent appearance and semantic forgetting. Code is available at https://github.com/jingyzhang/CGR.