Improving Replay-Based Continual Semantic Segmentation with Smart Data Selection

TL;DR

This paper explores replay strategies in continual semantic segmentation, highlighting the importance of balanced data selection to reduce catastrophic forgetting across class and domain incremental tasks.

Contribution

It introduces and evaluates novel replay data selection strategies tailored for semantic segmentation in continual learning, emphasizing class balance and feature-based sampling.

Findings

Uniform class distribution in buffers reduces bias towards new classes.

Sampling by feature distribution or median entropy improves domain adaptation.

Effective sampling decreases early layer representation shift, mitigating forgetting.

Abstract

Continual learning for Semantic Segmentation (CSS) is a rapidly emerging field, in which the capabilities of the segmentation model are incrementally improved by learning new classes or new domains. A central challenge in Continual Learning is overcoming the effects of catastrophic forgetting, which refers to the sudden drop in accuracy on previously learned tasks after the model is trained on new classes or domains. In continual classification this challenge is often overcome by replaying a small selection of samples from previous tasks, however replay is rarely considered in CSS. Therefore, we investigate the influences of various replay strategies for semantic segmentation and evaluate them in class- and domain-incremental settings. Our findings suggest that in a class-incremental setting, it is critical to achieve a uniform distribution for the different classes in the buffer to avoid a bias towards newly learned classes. In the domain-incremental setting, it is most effective to select buffer samples by uniformly sampling from the distribution of learned feature representations or by choosing samples with median entropy. Finally, we observe that the effective sampling methods help to decrease the representation shift significantly in early layers, which is a major cause of forgetting in domain-incremental learning.