PLOP: Learning without Forgetting for Continual Semantic Segmentation

TL;DR

PLOP introduces a novel continual learning method for semantic segmentation that effectively mitigates catastrophic forgetting by using multi-scale pooling distillation and entropy-based pseudo-labeling, outperforming existing approaches.

Contribution

The paper presents PLOP, a new continual learning framework that preserves spatial relationships and addresses background shift in semantic segmentation tasks.

Findings

PLOP outperforms state-of-the-art methods in existing CSS benchmarks.

It effectively mitigates catastrophic forgetting in continual semantic segmentation.

The approach demonstrates robustness on newly proposed challenging benchmarks.

Abstract

Deep learning approaches are nowadays ubiquitously used to tackle computer vision tasks such as semantic segmentation, requiring large datasets and substantial computational power. Continual learning for semantic segmentation (CSS) is an emerging trend that consists in updating an old model by sequentially adding new classes. However, continual learning methods are usually prone to catastrophic forgetting. This issue is further aggravated in CSS where, at each step, old classes from previous iterations are collapsed into the background. In this paper, we propose Local POD, a multi-scale pooling distillation scheme that preserves long- and short-range spatial relationships at feature level. Furthermore, we design an entropy-based pseudo-labelling of the background w.r.t. classes predicted by the old model to deal with background shift and avoid catastrophic forgetting of the old classes. Our approach, called PLOP, significantly outperforms state-of-the-art methods in existing CSS scenarios, as well as in newly proposed challenging benchmarks.