SEG-SAM: Semantic-Guided SAM for Unified Medical Image Segmentation

TL;DR

SEG-SAM introduces a semantic-guided approach to improve unified medical image segmentation by integrating semantic knowledge and a specialized decoder, outperforming existing SAM-based methods in medical tasks.

Contribution

The paper proposes SEG-SAM, a novel unified medical segmentation model that incorporates semantic medical knowledge and a semantic-aware decoder to enhance segmentation accuracy.

Findings

SEG-SAM outperforms state-of-the-art SAM-based methods in medical segmentation tasks.

The semantic-aware decoder improves semantic understanding and segmentation quality.

Cross-mask spatial alignment enhances mask overlap and prediction consistency.

Abstract

Recently, developing unified medical image segmentation models gains increasing attention, especially with the advent of the Segment Anything Model (SAM). SAM has shown promising binary segmentation performance in natural domains, however, transferring it to the medical domain remains challenging, as medical images often possess substantial inter-category overlaps. To address this, we propose the SEmantic-Guided SAM (SEG-SAM), a unified medical segmentation model that incorporates semantic medical knowledge to enhance medical segmentation performance. First, to avoid the potential conflict between binary and semantic predictions, we introduce a semantic-aware decoder independent of SAM's original decoder, specialized for both semantic segmentation on the prompted object and classification on unprompted objects in images. To further enhance the model's semantic understanding, we solicit key characteristics of medical categories from large language models and incorporate them into SEG-SAM through a text-to-vision semantic module, adaptively transferring the language information into the visual segmentation task. In the end, we introduce the cross-mask spatial alignment strategy to encourage greater overlap between the predicted masks from SEG-SAM's two decoders, thereby benefiting both predictions. Extensive experiments demonstrate that SEG-SAM outperforms state-of-the-art SAM-based methods in unified binary medical segmentation and task-specific methods in semantic medical segmentation, showcasing promising results and potential for broader medical applications.