Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations

TL;DR

This paper introduces a novel continual learning framework for semantic segmentation that reduces forgetting and improves recognition of new classes by shaping the latent space with prototypes matching, sparsification, and contrastive learning.

Contribution

It proposes a new latent space shaping approach combining prototypes matching, sparsification, and contrastive learning for class incremental semantic segmentation.

Findings

Significantly outperforms state-of-the-art methods on Pascal VOC2012.

Effectively reduces catastrophic forgetting in continual learning.

Improves recognition accuracy for novel classes.

Abstract

Deep neural networks suffer from the major limitation of catastrophic forgetting old tasks when learning new ones. In this paper we focus on class incremental continual learning in semantic segmentation, where new categories are made available over time while previous training data is not retained. The proposed continual learning scheme shapes the latent space to reduce forgetting whilst improving the recognition of novel classes. Our framework is driven by three novel components which we also combine on top of existing techniques effortlessly. First, prototypes matching enforces latent space consistency on old classes, constraining the encoder to produce similar latent representation for previously seen classes in the subsequent steps. Second, features sparsification allows to make room in the latent space to accommodate novel classes. Finally, contrastive learning is employed to cluster features according to their semantics while tearing apart those of different classes. Extensive evaluation on the Pascal VOC2012 and ADE20K datasets demonstrates the effectiveness of our approach, significantly outperforming state-of-the-art methods.