Strike a Balance in Continual Panoptic Segmentation

TL;DR

This paper introduces BalConpas, a novel method for continual panoptic segmentation that balances stability and plasticity through past-class distillation, class-proportional memory, and anti-misguidance losses, achieving superior results on ADE20K.

Contribution

It proposes new techniques for continual panoptic segmentation, including past-class backtrace distillation, class-proportional memory, and balanced anti-misguidance losses, to improve knowledge retention and adaptation.

Findings

Outperforms existing methods on ADE20K dataset

Effectively balances stability and plasticity in segmentation

Enhances recall of past classes during continual learning

Abstract

This study explores the emerging area of continual panoptic segmentation, highlighting three key balances. First, we introduce past-class backtrace distillation to balance the stability of existing knowledge with the adaptability to new information. This technique retraces the features associated with past classes based on the final label assignment results, performing knowledge distillation targeting these specific features from the previous model while allowing other features to flexibly adapt to new information. Additionally, we introduce a class-proportional memory strategy, which aligns the class distribution in the replay sample set with that of the historical training data. This strategy maintains a balanced class representation during replay, enhancing the utility of the limited-capacity replay sample set in recalling prior classes. Moreover, recognizing that replay samples are annotated only for the classes of their original step, we devise balanced anti-misguidance losses, which combat the impact of incomplete annotations without incurring classification bias. Building upon these innovations, we present a new method named Balanced Continual Panoptic Segmentation (BalConpas). Our evaluation on the challenging ADE20K dataset demonstrates its superior performance compared to existing state-of-the-art methods. The official code is available at https://github.com/jinpeng0528/BalConpas.