Continual Learning for Class- and Domain-Incremental Semantic Segmentation

TL;DR

This paper evaluates and adapts continual learning methods for semantic segmentation, focusing on class- and domain-incremental scenarios, and establishes baseline protocols and insights into effective strategies for these tasks.

Contribution

It introduces evaluation protocols for continual semantic segmentation and analyzes the effectiveness of knowledge distillation and replay methods in different incremental settings.

Findings

Knowledge distillation is effective in class-incremental segmentation.

Replay methods outperform others in domain-incremental segmentation.

Semantic segmentation requires different continual learning strategies than image classification.

Abstract

The field of continual deep learning is an emerging field and a lot of progress has been made. However, concurrently most of the approaches are only tested on the task of image classification, which is not relevant in the field of intelligent vehicles. Only recently approaches for class-incremental semantic segmentation were proposed. However, all of those approaches are based on some form of knowledge distillation. At the moment there are no investigations on replay-based approaches that are commonly used for object recognition in a continual setting. At the same time while unsupervised domain adaption for semantic segmentation gained a lot of traction, investigations regarding domain-incremental learning in an continual setting is not well-studied. Therefore, the goal of our work is to evaluate and adapt established solutions for continual object recognition to the task of semantic segmentation and to provide baseline methods and evaluation protocols for the task of continual semantic segmentation. We firstly introduce evaluation protocols for the class- and domain-incremental segmentation and analyze selected approaches. We show that the nature of the task of semantic segmentation changes which methods are most effective in mitigating forgetting compared to image classification. Especially, in class-incremental learning knowledge distillation proves to be a vital tool, whereas in domain-incremental learning replay methods are the most effective method.