Unlocking the Power of SAM 2 for Few-Shot Segmentation

TL;DR

This paper leverages SAM 2's video segmentation capabilities to improve few-shot segmentation by designing a pseudo prompt generator, iterative memory refinement, and support-calibrated attention, achieving significant performance gains.

Contribution

It introduces a novel framework that adapts SAM 2 for FSS by addressing identity mismatch and memory accuracy issues with new modules, enhancing segmentation performance.

Findings

1-shot mIoU improved by 4.2% over baseline

Effective handling of query background features

Validated on PASCAL-5i and COCO-20i datasets

Abstract

Few-Shot Segmentation (FSS) aims to learn class-agnostic segmentation on few classes to segment arbitrary classes, but at the risk of overfitting. To address this, some methods use the well-learned knowledge of foundation models (e.g., SAM) to simplify the learning process. Recently, SAM 2 has extended SAM by supporting video segmentation, whose class-agnostic matching ability is useful to FSS. A simple idea is to encode support foreground (FG) features as memory, with which query FG features are matched and fused. Unfortunately, the FG objects in different frames of SAM 2's video data are always the same identity, while those in FSS are different identities, i.e., the matching step is incompatible. Therefore, we design Pseudo Prompt Generator to encode pseudo query memory, matching with query features in a compatible way. However, the memories can never be as accurate as the real ones, i.e., they are likely to contain incomplete query FG, and some unexpected query background (BG) features, leading to wrong segmentation. Hence, we further design Iterative Memory Refinement to fuse more query FG features into the memory, and devise a Support-Calibrated Memory Attention to suppress the unexpected query BG features in memory. Extensive experiments have been conducted on PASCAL-5$^i$ and COCO-20$^i$ to validate the effectiveness of our design, e.g., the 1-shot mIoU can be 4.2% better than the best baseline.