Fully Automated SAM for Single-source Domain Generalization in Medical Image Segmentation

TL;DR

This paper introduces FA-SAM, a fully automated framework for medical image segmentation that enhances SAM's domain generalization by automatically generating prompts and integrating multi-scale features, improving accuracy across different datasets.

Contribution

The paper presents FA-SAM, a novel framework that automates prompt generation and fuses multi-scale features to improve SAM's performance in medical image segmentation across domains.

Findings

FA-SAM outperforms existing methods on prostate and fundus datasets.

The AGM and IPEF modules significantly improve segmentation accuracy.

FA-SAM demonstrates robustness to poor prompts and domain shifts.

Abstract

Although SAM-based single-source domain generalization models for medical image segmentation can mitigate the impact of domain shift on the model in cross-domain scenarios, these models still face two major challenges. First, the segmentation of SAM is highly dependent on domain-specific expert-annotated prompts, which prevents SAM from achieving fully automated medical image segmentation and therefore limits its application in clinical settings. Second, providing poor prompts (such as bounding boxes that are too small or too large) to the SAM prompt encoder can mislead SAM into generating incorrect mask results. Therefore, we propose the FA-SAM, a single-source domain generalization framework for medical image segmentation that achieves fully automated SAM. FA-SAM introduces two key innovations: an Auto-prompted Generation Model (AGM) branch equipped with a Shallow Feature Uncertainty Modeling (SUFM) module, and an Image-Prompt Embedding Fusion (IPEF) module integrated into the SAM mask decoder. Specifically, AGM models the uncertainty distribution of shallow features through the SUFM module to generate bounding box prompts for the target domain, enabling fully automated segmentation with SAM. The IPEF module integrates multiscale information from SAM image embeddings and prompt embeddings to capture global and local details of the target object, enabling SAM to mitigate the impact of poor prompts. Extensive experiments on publicly available prostate and fundus vessel datasets validate the effectiveness of FA-SAM and highlight its potential to address the above challenges.