Foreground-Covering Prototype Generation and Matching for SAM-Aided Few-Shot Segmentation

TL;DR

This paper introduces a novel method for few-shot segmentation that leverages SAM and ResNet features with prototype matching and iterative cross-attention, achieving state-of-the-art results.

Contribution

It proposes a foreground-covering prototype generation and matching approach that effectively utilizes SAM and ResNet features with attention-guided pseudo-masks for improved FSS.

Findings

Achieves state-of-the-art performance on multiple datasets.

Effectively utilizes attention-based pseudo-masks for segmentation.

Demonstrates the benefit of combining SAM and ResNet features.

Abstract

We propose Foreground-Covering Prototype Generation and Matching to resolve Few-Shot Segmentation (FSS), which aims to segment target regions in unlabeled query images based on labeled support images. Unlike previous research, which typically estimates target regions in the query using support prototypes and query pixels, we utilize the relationship between support and query prototypes. To achieve this, we utilize two complementary features: SAM Image Encoder features for pixel aggregation and ResNet features for class consistency. Specifically, we construct support and query prototypes with SAM features and distinguish query prototypes of target regions based on ResNet features. For the query prototype construction, we begin by roughly guiding foreground regions within SAM features using the conventional pseudo-mask, then employ iterative cross-attention to aggregate foreground features into learnable tokens. Here, we discover that the cross-attention weights can effectively alternate the conventional pseudo-mask. Therefore, we use the attention-based pseudo-mask to guide ResNet features to focus on the foreground, then infuse the guided ResNet feature into the learnable tokens to generate class-consistent query prototypes. The generation of the support prototype is conducted symmetrically to that of the query one, with the pseudo-mask replaced by the ground-truth mask. Finally, we compare these query prototypes with support ones to generate prompts, which subsequently produce object masks through the SAM Mask Decoder. Our state-of-the-art performances on various datasets validate the effectiveness of the proposed method for FSS. Our official code is available at https://github.com/SuhoPark0706/FCP