Continual Semantic Segmentation with Automatic Memory Sample Selection

TL;DR

This paper introduces an automatic, comprehensive memory sample selection method for continual semantic segmentation, significantly improving performance by effectively choosing informative samples for replay.

Contribution

It proposes a novel automatic sample selection mechanism that learns an optimal policy considering diversity and class performance, outperforming existing strategies.

Findings

Achieves state-of-the-art results on Pascal-VOC 2012 and ADE 20K datasets.

Outperforms previous methods by 12.54% on Pascal-VOC 2012 6-stage setting.

Demonstrates the effectiveness of learned selection policies in continual learning.

Abstract

Continual Semantic Segmentation (CSS) extends static semantic segmentation by incrementally introducing new classes for training. To alleviate the catastrophic forgetting issue in CSS, a memory buffer that stores a small number of samples from the previous classes is constructed for replay. However, existing methods select the memory samples either randomly or based on a single-factor-driven handcrafted strategy, which has no guarantee to be optimal. In this work, we propose a novel memory sample selection mechanism that selects informative samples for effective replay in a fully automatic way by considering comprehensive factors including sample diversity and class performance. Our mechanism regards the selection operation as a decision-making process and learns an optimal selection policy that directly maximizes the validation performance on a reward set. To facilitate the selection decision, we design a novel state representation and a dual-stage action space. Our extensive experiments on Pascal-VOC 2012 and ADE 20K datasets demonstrate the effectiveness of our approach with state-of-the-art (SOTA) performance achieved, outperforming the second-place one by 12.54% for the 6stage setting on Pascal-VOC 2012.