Enhancing Generalized Few-Shot Semantic Segmentation via Effective Knowledge Transfer

TL;DR

This paper introduces a novel knowledge transfer framework for generalized few-shot semantic segmentation, employing prototype modulation, classifier calibration, and context consistency to improve segmentation accuracy for both base and novel classes.

Contribution

It proposes new modules for effective knowledge transfer and context learning, significantly improving GFSS performance over existing methods.

Findings

Achieves state-of-the-art results on PASCAL-5i and COCO-20i datasets.

Demonstrates the effectiveness of prototype modulation and classifier calibration.

Shows substantial performance gains in generalized few-shot segmentation tasks.

Abstract

Generalized few-shot semantic segmentation (GFSS) aims to segment objects of both base and novel classes, using sufficient samples of base classes and few samples of novel classes. Representative GFSS approaches typically employ a two-phase training scheme, involving base class pre-training followed by novel class fine-tuning, to learn the classifiers for base and novel classes respectively. Nevertheless, distribution gap exists between base and novel classes in this process. To narrow this gap, we exploit effective knowledge transfer from base to novel classes. First, a novel prototype modulation module is designed to modulate novel class prototypes by exploiting the correlations between base and novel classes. Second, a novel classifier calibration module is proposed to calibrate the weight distribution of the novel classifier according to that of the base classifier. Furthermore, existing GFSS approaches suffer from a lack of contextual information for novel classes due to their limited samples, we thereby introduce a context consistency learning scheme to transfer the contextual knowledge from base to novel classes. Extensive experiments on PASCAL-5$^i$ and COCO-20$^i$ demonstrate that our approach significantly enhances the state of the art in the GFSS setting. The code is available at: https://github.com/HHHHedy/GFSS-EKT.