Modeling the Background for Incremental and Weakly-Supervised Semantic Segmentation

TL;DR

This paper presents a new incremental learning approach for semantic segmentation that addresses background semantic shift and weak supervision, significantly improving performance on multiple datasets.

Contribution

It introduces a novel loss function and classifier initialization strategy to handle background shift and extends the method to weakly supervised settings.

Findings

Outperforms state-of-the-art on Pascal-VOC, ADE20K, Cityscapes

Effectively manages background semantic shift in incremental learning

Extends to weakly supervised segmentation with partial annotations

Abstract

Deep neural networks have enabled major progresses in semantic segmentation. However, even the most advanced neural architectures suffer from important limitations. First, they are vulnerable to catastrophic forgetting, i.e. they perform poorly when they are required to incrementally update their model as new classes are available. Second, they rely on large amount of pixel-level annotations to produce accurate segmentation maps. To tackle these issues, we introduce a novel incremental class learning approach for semantic segmentation taking into account a peculiar aspect of this task: since each training step provides annotation only for a subset of all possible classes, pixels of the background class exhibit a semantic shift. Therefore, we revisit the traditional distillation paradigm by designing novel loss terms which explicitly account for the background shift. Additionally, we introduce a novel strategy to initialize classifier's parameters at each step in order to prevent biased predictions toward the background class. Finally, we demonstrate that our approach can be extended to point- and scribble-based weakly supervised segmentation, modeling the partial annotations to create priors for unlabeled pixels. We demonstrate the effectiveness of our approach with an extensive evaluation on the Pascal-VOC, ADE20K, and Cityscapes datasets, significantly outperforming state-of-the-art methods.