Sam2Rad: A Segmentation Model for Medical Images with Learnable Prompts

TL;DR

Sam2Rad introduces a prompt learning method that adapts foundation models like SAM for ultrasound bone segmentation, eliminating manual prompts and achieving significant performance improvements across multiple datasets.

Contribution

The paper presents Sam2Rad, a novel prompt predictor network that enables automatic, efficient adaptation of SAM for US image segmentation without manual prompts.

Findings

Improves Dice scores by 2-7% on US datasets.

Achieves up to 33% improvement in shoulder segmentation.

Effective with as few as 10 labeled images.

Abstract

Foundation models like the segment anything model require high-quality manual prompts for medical image segmentation, which is time-consuming and requires expertise. SAM and its variants often fail to segment structures in ultrasound (US) images due to domain shift. We propose Sam2Rad, a prompt learning approach to adapt SAM and its variants for US bone segmentation without human prompts. It introduces a prompt predictor network (PPN) with a cross-attention module to predict prompt embeddings from image encoder features. PPN outputs bounding box and mask prompts, and 256-dimensional embeddings for regions of interest. The framework allows optional manual prompting and can be trained end-to-end using parameter-efficient fine-tuning (PEFT). Sam2Rad was tested on 3 musculoskeletal US datasets: wrist (3822 images), rotator cuff (1605 images), and hip (4849 images). It improved performance across all datasets without manual prompts, increasing Dice scores by 2-7% for hip/wrist and up to 33% for shoulder data. Sam2Rad can be trained with as few as 10 labeled images and is compatible with any SAM architecture for automatic segmentation.