Towards Realistic Incremental Scenario in Class Incremental Semantic Segmentation

TL;DR

This paper proposes a more realistic incremental learning scenario for semantic segmentation, addressing flaws in previous setups, and introduces a new baseline that achieves state-of-the-art results in class-incremental tasks.

Contribution

It introduces the partitioned scenario for more practical incremental learning, corrects data retrieval issues, and presents a competitive memory-based baseline for improved performance.

Findings

The partitioned scenario better reflects real-world incremental learning.

Corrected data retrieval improves baseline performance.

MiB-AugM achieves state-of-the-art results across multiple class-incremental tasks.

Abstract

This paper addresses the unrealistic aspect of the commonly adopted Continuous Incremental Semantic Segmentation (CISS) scenario, termed overlapped. We point out that overlapped allows the same image to reappear in future tasks with different pixel labels, which is far from practical incremental learning scenarios. Moreover, we identified that this flawed scenario may lead to biased results for two commonly used techniques in CISS, pseudo-labeling and exemplar memory, resulting in unintended advantages or disadvantages for certain techniques. To mitigate this, a practical scenario called partitioned is proposed, in which the dataset is first divided into distinct subsets representing each class, and then the subsets are assigned to each corresponding task. This efficiently addresses the issue above while meeting the requirement of CISS scenario, such as capturing the background shifts. Furthermore, we identify and address the code implementation issues related to retrieving data from the exemplar memory, which was ignored in previous works. Lastly, we introduce a simple yet competitive memory-based baseline, MiB-AugM, that handles background shifts of current tasks in the exemplar memory. This baseline achieves state-of-the-art results across multiple tasks involving learning numerous new classes.