RBC: Rectifying the Biased Context in Continual Semantic Segmentation

TL;DR

This paper introduces a novel framework for continual semantic segmentation that addresses context bias and class imbalance, significantly improving performance over existing methods.

Contribution

It proposes a biased-context-rectified framework with a context-rectified learning scheme, a consistency loss, and an adaptive re-weighting strategy for better CSS performance.

Findings

Outperforms state-of-the-art CSS methods by a large margin

Effectively mitigates old-class forgetting and new-class overfitting

Enhances segmentation accuracy in continual learning scenarios

Abstract

Recent years have witnessed a great development of Convolutional Neural Networks in semantic segmentation, where all classes of training images are simultaneously available. In practice, new images are usually made available in a consecutive manner, leading to a problem called Continual Semantic Segmentation (CSS). Typically, CSS faces the forgetting problem since previous training images are unavailable, and the semantic shift problem of the background class. Considering the semantic segmentation as a context-dependent pixel-level classification task, we explore CSS from a new perspective of context analysis in this paper. We observe that the context of old-class pixels in the new images is much more biased on new classes than that in the old images, which can sharply aggravate the old-class forgetting and new-class overfitting. To tackle the obstacle, we propose a biased-context-rectified CSS framework with a context-rectified image-duplet learning scheme and a biased-context-insensitive consistency loss. Furthermore, we propose an adaptive re-weighting class-balanced learning strategy for the biased class distribution. Our approach outperforms state-of-the-art methods by a large margin in existing CSS scenarios.