Task Consistent Prototype Learning for Incremental Few-shot Semantic Segmentation

TL;DR

This paper proposes a meta-learning-based prototype method for incremental few-shot semantic segmentation, enabling models to adapt quickly to new classes with minimal forgetting by simulating incremental tasks during training.

Contribution

It introduces a novel meta-learning framework with prototype space redistribution to improve incremental learning in semantic segmentation tasks.

Findings

Outperforms existing methods on PASCAL and COCO benchmarks

Effectively balances learning new classes and retaining old knowledge

Provides insights into incremental few-shot segmentation challenges

Abstract

Incremental Few-Shot Semantic Segmentation (iFSS) tackles a task that requires a model to continually expand its segmentation capability on novel classes using only a few annotated examples. Typical incremental approaches encounter a challenge that the objective of the base training phase (fitting base classes with sufficient instances) does not align with the incremental learning phase (rapidly adapting to new classes with less forgetting). This disconnect can result in suboptimal performance in the incremental setting. This study introduces a meta-learning-based prototype approach that encourages the model to learn how to adapt quickly while preserving previous knowledge. Concretely, we mimic the incremental evaluation protocol during the base training session by sampling a sequence of pseudo-incremental tasks. Each task in the simulated sequence is trained using a meta-objective to enable rapid adaptation without forgetting. To enhance discrimination among class prototypes, we introduce prototype space redistribution learning, which dynamically updates class prototypes to establish optimal inter-prototype boundaries within the prototype space. Extensive experiments on iFSS datasets built upon PASCAL and COCO benchmarks show the advanced performance of the proposed approach, offering valuable insights for addressing iFSS challenges.