VesSAM: Efficient Multi-Prompting for Segmenting Complex Vessel

TL;DR

VesSAM is a tailored, efficient multi-prompting framework for 2D vessel segmentation that outperforms existing models in accuracy, generalizes well to new data, and reduces computational complexity.

Contribution

The paper introduces VesSAM, a novel vessel segmentation framework combining local texture enhancement, anatomical prompts, and a lightweight decoder, with an automated multi-prompt annotation pipeline.

Findings

VesSAM outperforms state-of-the-art PEFT-based SAM variants by over 10% Dice.

VesSAM achieves over 13% higher IoU compared to baselines.

VesSAM generalizes effectively to out-of-distribution datasets.

Abstract

Accurate vessel segmentation is critical for clinical applications such as disease diagnosis and surgical planning, yet remains challenging due to thin, branching structures and low texture contrast. While foundation models like the Segment Anything Model (SAM) have shown promise in generic segmentation, they perform sub-optimally on vascular structures. In this work, we present VesSAM, a powerful and efficient framework tailored for 2D vessel segmentation. VesSAM integrates (1) a convolutional adapter to enhance local texture features, (2) a multi-prompt encoder that fuses anatomical prompts, including skeletons, bifurcation points, and segment midpoints, via hierarchical cross-attention, and (3) a lightweight mask decoder to reduce jagged artifacts. We also introduce an automated pipeline to generate structured multi-prompt annotations, and curate a diverse benchmark dataset spanning 8 datasets across 5 imaging modalities. Experimental results demonstrate that VesSAM consistently outperforms state-of-the-art PEFT-based SAM variants by over 10% Dice and 13% IoU, and achieves competitive performance compared to fully fine-tuned methods, with significantly fewer parameters. VesSAM also generalizes well to out-of-distribution (OoD) settings, outperforming all baselines in average OoD Dice and IoU.