Semantically Meaningful Class Prototype Learning for One-Shot Image Semantic Segmentation

TL;DR

This paper introduces a novel one-shot semantic segmentation method that leverages multi-class label information, pyramid feature fusion, and prototype guidance to improve segmentation accuracy for new classes with only one annotated image.

Contribution

The paper proposes a multi-class label utilization strategy, a pyramid feature fusion module, and a self-prototype guidance branch for more accurate one-shot segmentation.

Findings

Outperforms existing methods on benchmark datasets.

Effectively utilizes multi-class labels during training.

Produces more compact and robust class prototypes.

Abstract

One-shot semantic image segmentation aims to segment the object regions for the novel class with only one annotated image. Recent works adopt the episodic training strategy to mimic the expected situation at testing time. However, these existing approaches simulate the test conditions too strictly during the training process, and thus cannot make full use of the given label information. Besides, these approaches mainly focus on the foreground-background target class segmentation setting. They only utilize binary mask labels for training. In this paper, we propose to leverage the multi-class label information during the episodic training. It will encourage the network to generate more semantically meaningful features for each category. After integrating the target class cues into the query features, we then propose a pyramid feature fusion module to mine the fused features for the final classifier. Furthermore, to take more advantage of the support image-mask pair, we propose a self-prototype guidance branch to support image segmentation. It can constrain the network for generating more compact features and a robust prototype for each semantic class. For inference, we propose a fused prototype guidance branch for the segmentation of the query image. Specifically, we leverage the prediction of the query image to extract the pseudo-prototype and combine it with the initial prototype. Then we utilize the fused prototype to guide the final segmentation of the query image. Extensive experiments demonstrate the superiority of our proposed approach.