Adaptive Prompt Learning with SAM for Few-shot Scanning Probe Microscope Image Segmentation

TL;DR

This paper introduces APL-SAM, a novel framework that adapts the Segment Anything Model for few-shot scanning probe microscope image segmentation, significantly improving accuracy with minimal data and eliminating extensive user prompts.

Contribution

The paper proposes an adaptive prompt learning module and a multi-source mask decoder tailored for few-shot SPM image segmentation, along with a new dataset, SPM-Seg.

Findings

Achieves over 30% improvement in Dice coefficient with one-shot guidance

Outperforms original SAM and state-of-the-art few-shot methods

Surpasses fully supervised approaches in SPM segmentation accuracy

Abstract

The Segment Anything Model (SAM) has demonstrated strong performance in image segmentation of natural scene images. However, its effectiveness diminishes markedly when applied to specific scientific domains, such as Scanning Probe Microscope (SPM) images. This decline in accuracy can be attributed to the distinct data distribution and limited availability of the data inherent in the scientific images. On the other hand, the acquisition of adequate SPM datasets is both time-intensive and laborious as well as skill-dependent. To address these challenges, we propose an Adaptive Prompt Learning with SAM (APL-SAM) framework tailored for few-shot SPM image segmentation. Our approach incorporates two key innovations to enhance SAM: 1) An Adaptive Prompt Learning module leverages few-shot embeddings derived from limited support set to learn adaptively central representatives, serving as visual prompts. This innovation eliminates the need for time-consuming online user interactions for providing prompts, such as exhaustively marking points and bounding boxes slice by slice; 2) A multi-source, multi-level mask decoder specifically designed for few-shot SPM image segmentation is introduced, which can effectively capture the correspondence between the support and query images. To facilitate comprehensive training and evaluation, we introduce a new dataset, SPM-Seg, curated for SPM image segmentation. Extensive experiments on this dataset reveal that the proposed APL-SAM framework significantly outperforms the original SAM, achieving over a 30% improvement in terms of Dice Similarity Coefficient with only one-shot guidance. Moreover, APL-SAM surpasses state-of-the-art few-shot segmentation methods and even fully supervised approaches in performance. Code and dataset used in this study will be made available upon acceptance.