CycleSAM: Few-Shot Surgical Scene Segmentation with Cycle- and Scene-Consistent Feature Matching

TL;DR

CycleSAM is a novel few-shot surgical scene segmentation method that uses cycle- and scene-consistent feature matching, significantly improving performance on surgical datasets by addressing domain gaps and leveraging self-supervised features.

Contribution

It introduces a data-efficient training approach with soft constraints for robust feature matching, specifically tailored for surgical image segmentation.

Findings

Outperforms existing few-shot SAM methods by 2-4x in accuracy.

Achieves strong results over traditional adaptation methods.

Effective in diverse surgical datasets.

Abstract

Surgical image segmentation is highly challenging, primarily due to scarcity of annotated data. Generalist prompted segmentation models like the Segment-Anything Model (SAM) can help tackle this task, but because they require image-specific visual prompts for effective performance, their use is limited to improving data annotation efficiency. Recent approaches extend SAM to automatic segmentation by using a few labeled reference images to predict point prompts; however, they rely on feature matching pipelines that lack robustness to out-of-domain data like surgical images. To tackle this problem, we introduce CycleSAM, an improved visual prompt learning approach that employs a data-efficient training phase and enforces a series of soft constraints to produce high-quality feature similarity maps. CycleSAM label-efficiently addresses domain gap by leveraging surgery-specific self-supervised feature extractors, then adapts the resulting features through a short parameter-efficient training stage, enabling it to produce informative similarity maps. CycleSAM further filters the similarity maps with a series of consistency constraints before robustly sampling diverse point prompts for each object instance. In our experiments on four diverse surgical datasets, we find that CycleSAM outperforms existing few-shot SAM approaches by a factor of 2-4x in both 1-shot and 5-shot settings, while also achieving strong performance gains over traditional linear probing, parameter-efficient adaptation, and pseudo-labeling methods.