Gradient-Semantic Compensation for Incremental Semantic Segmentation

TL;DR

This paper introduces Gradient-Semantic Compensation (GSC), a novel approach for incremental semantic segmentation that addresses catastrophic forgetting and background shift by balancing gradient updates and distilling semantic relations.

Contribution

The paper proposes a combined gradient and semantic perspective model with gradient compensation, semantic relation distillation, and pseudo re-labeling for improved incremental segmentation.

Findings

Effective in reducing catastrophic forgetting.

Improves segmentation accuracy on multiple datasets.

Outperforms existing incremental segmentation methods.

Abstract

Incremental semantic segmentation aims to continually learn the segmentation of new coming classes without accessing the training data of previously learned classes. However, most current methods fail to address catastrophic forgetting and background shift since they 1) treat all previous classes equally without considering different forgetting paces caused by imbalanced gradient back-propagation; 2) lack strong semantic guidance between classes. To tackle the above challenges, in this paper, we propose a Gradient-Semantic Compensation (GSC) model, which surmounts incremental semantic segmentation from both gradient and semantic perspectives. Specifically, to address catastrophic forgetting from the gradient aspect, we develop a step-aware gradient compensation that can balance forgetting paces of previously seen classes via re-weighting gradient backpropagation. Meanwhile, we propose a soft-sharp semantic relation distillation to distill consistent inter-class semantic relations via soft labels for alleviating catastrophic forgetting from the semantic aspect. In addition, we develop a prototypical pseudo re-labeling that provides strong semantic guidance to mitigate background shift. It produces high-quality pseudo labels for old classes in the background by measuring distances between pixels and class-wise prototypes. Extensive experiments on three public datasets, i.e., Pascal VOC 2012, ADE20K, and Cityscapes, demonstrate the effectiveness of our proposed GSC model.