Adaptive Prototype Replay for Class Incremental Semantic Segmentation

TL;DR

This paper introduces Adaptive Prototype Replay (Adapter), a novel method for class incremental semantic segmentation that dynamically updates class prototypes to match evolving representations, significantly improving continual learning performance.

Contribution

The paper proposes an adaptive prototype update strategy and new loss functions to better align prototypes with updated class representations in CISS, addressing limitations of fixed prototypes.

Findings

Achieves state-of-the-art results on Pascal VOC and ADE20K datasets.

Effectively reduces catastrophic forgetting in multi-step CISS tasks.

Demonstrates robustness across various incremental learning scenarios.

Abstract

Class incremental semantic segmentation (CISS) aims to segment new classes during continual steps while preventing the forgetting of old knowledge. Existing methods alleviate catastrophic forgetting by replaying distributions of previously learned classes using stored prototypes or features. However, they overlook a critical issue: in CISS, the representation of class knowledge is updated continuously through incremental learning, whereas prototype replay methods maintain fixed prototypes. This mismatch between updated representation and fixed prototypes limits the effectiveness of the prototype replay strategy. To address this issue, we propose the Adaptive prototype replay (Adapter) for CISS in this paper. Adapter comprises an adaptive deviation compen sation (ADC) strategy and an uncertainty-aware constraint (UAC) loss. Specifically, the ADC strategy dynamically updates the stored prototypes based on the estimated representation shift distance to match the updated representation of old class. The UAC loss reduces prediction uncertainty, aggregating discriminative features to aid in generating compact prototypes. Additionally, we introduce a compensation-based prototype similarity discriminative (CPD) loss to ensure adequate differentiation between similar prototypes, thereby enhancing the efficiency of the adaptive prototype replay strategy. Extensive experiments on Pascal VOC and ADE20K datasets demonstrate that Adapter achieves state-of-the-art results and proves effective across various CISS tasks, particularly in challenging multi-step scenarios. The code and model is available at https://github.com/zhu-gl-ux/Adapter.