CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation

TL;DR

The paper introduces CAT, a novel model that combines visual and textual prompts to improve multi-organ and tumor segmentation in medical images, addressing the limitations of existing prompt-based methods.

Contribution

It proposes a dual-prompt schema and a unified framework with a ShareRefiner to enhance segmentation accuracy across diverse medical imaging scenarios.

Findings

Superior performance on 10 public CT datasets

Effective segmentation of tumors across multiple cancer stages

Demonstrates the benefits of multimodal prompt coordination

Abstract

Existing promptable segmentation methods in the medical imaging field primarily consider either textual or visual prompts to segment relevant objects, yet they often fall short when addressing anomalies in medical images, like tumors, which may vary greatly in shape, size, and appearance. Recognizing the complexity of medical scenarios and the limitations of textual or visual prompts, we propose a novel dual-prompt schema that leverages the complementary strengths of visual and textual prompts for segmenting various organs and tumors. Specifically, we introduce CAT, an innovative model that Coordinates Anatomical prompts derived from 3D cropped images with Textual prompts enriched by medical domain knowledge. The model architecture adopts a general query-based design, where prompt queries facilitate segmentation queries for mask prediction. To synergize two types of prompts within a unified framework, we implement a ShareRefiner, which refines both segmentation and prompt queries while disentangling the two types of prompts. Trained on a consortium of 10 public CT datasets, CAT demonstrates superior performance in multiple segmentation tasks. Further validation on a specialized in-house dataset reveals the remarkable capacity of segmenting tumors across multiple cancer stages. This approach confirms that coordinating multimodal prompts is a promising avenue for addressing complex scenarios in the medical domain.