SpinalSAM-R1: A Vision-Language Multimodal Interactive System for Spine CT Segmentation

TL;DR

SpinalSAM-R1 is a multimodal interactive system that combines vision and language models to improve spine CT segmentation, enabling natural language-guided refinement and achieving high accuracy and efficiency in clinical settings.

Contribution

The paper introduces SpinalSAM-R1, a novel multimodal system integrating a fine-tuned SAM with DeepSeek-R1 for improved spine CT segmentation with interactive, language-guided refinement capabilities.

Findings

Achieves superior segmentation performance on spine CT images.

Supports 11 clinical operations with 94.3% parsing accuracy.

Provides sub-800 ms response times for interactive prompts.

Abstract

The anatomical structure segmentation of the spine and adjacent structures from computed tomography (CT) images is a key step for spinal disease diagnosis and treatment. However, the segmentation of CT images is impeded by low contrast and complex vertebral boundaries. Although advanced models such as the Segment Anything Model (SAM) have shown promise in various segmentation tasks, their performance in spinal CT imaging is limited by high annotation requirements and poor domain adaptability. To address these limitations, we propose SpinalSAM-R1, a multimodal vision-language interactive system that integrates a fine-tuned SAM with DeepSeek-R1, for spine CT image segmentation. Specifically, our SpinalSAM-R1 introduces an anatomy-guided attention mechanism to improve spine segmentation performance, and a semantics-driven interaction protocol powered by DeepSeek-R1, enabling natural language-guided refinement. The SpinalSAM-R1 is fine-tuned using Low-Rank Adaptation (LoRA) for efficient adaptation. We validate our SpinalSAM-R1 on the spine anatomical structure with CT images. Experimental results suggest that our method achieves superior segmentation performance. Meanwhile, we develop a PyQt5-based interactive software, which supports point, box, and text-based prompts. The system supports 11 clinical operations with 94.3\% parsing accuracy and sub-800 ms response times. The software is released on https://github.com/6jm233333/spinalsam-r1.