Prototypical Kernel Learning and Open-set Foreground Perception for Generalized Few-shot Semantic Segmentation

TL;DR

This paper introduces a novel approach combining prototypical kernel learning and open-set foreground perception to improve generalized few-shot semantic segmentation, effectively handling unseen classes and reducing embedding bias.

Contribution

It proposes a joint framework with learnable kernels and a foreground perception module, enhancing GFSS performance and enabling class incremental learning.

Findings

Outperforms previous state-of-the-art on PASCAL-5i and COCO-20i datasets.

Effectively detects open-set foreground and mitigates embedding prejudice.

Supports class incremental few-shot segmentation.

Abstract

Generalized Few-shot Semantic Segmentation (GFSS) extends Few-shot Semantic Segmentation (FSS) to simultaneously segment unseen classes and seen classes during evaluation. Previous works leverage additional branch or prototypical aggregation to eliminate the constrained setting of FSS. However, representation division and embedding prejudice, which heavily results in poor performance of GFSS, have not been synthetical considered. We address the aforementioned problems by jointing the prototypical kernel learning and open-set foreground perception. Specifically, a group of learnable kernels is proposed to perform segmentation with each kernel in charge of a stuff class. Then, we explore to merge the prototypical learning to the update of base-class kernels, which is consistent with the prototype knowledge aggregation of few-shot novel classes. In addition, a foreground contextual perception module cooperating with conditional bias based inference is adopted to perform class-agnostic as well as open-set foreground detection, thus to mitigate the embedding prejudice and prevent novel targets from being misclassified as background. Moreover, we also adjust our method to the Class Incremental Few-shot Semantic Segmentation (CIFSS) which takes the knowledge of novel classes in a incremental stream. Extensive experiments on PASCAL-5i and COCO-20i datasets demonstrate that our method performs better than previous state-of-the-art.