Text-Promptable Propagation for Referring Medical Image Sequence Segmentation

TL;DR

This paper introduces Text-Promptable Propagation (TPP), a novel model for segmenting anatomical structures in medical image sequences based on natural language, addressing limitations of existing models in tracking and guidance.

Contribution

The paper proposes TPP, a new cross-modal, Transformer-based model that explicitly tracks objects across medical image sequences using text prompts, and introduces a large-scale benchmark for this task.

Findings

TPP outperforms existing methods in medical segmentation.

TPP effectively tracks objects across sequences using text prompts.

The benchmark covers 4 modalities and 20 organs/lesions.

Abstract

Referring Medical Image Sequence Segmentation (Ref-MISS) is a novel and challenging task that aims to segment anatomical structures in medical image sequences (\emph{e.g.} endoscopy, ultrasound, CT, and MRI) based on natural language descriptions. This task holds significant clinical potential and offers a user-friendly advancement in medical imaging interpretation. Existing 2D and 3D segmentation models struggle to explicitly track objects of interest across medical image sequences, and lack support for nteractive, text-driven guidance. To address these limitations, we propose Text-Promptable Propagation (TPP), a model designed for referring medical image sequence segmentation. TPP captures the intrinsic relationships among sequential images along with their associated textual descriptions. Specifically, it enables the recognition of referred objects through cross-modal referring interaction, and maintains continuous tracking across the sequence via Transformer-based triple propagation, using text embeddings as queries. To support this task, we curate a large-scale benchmark, Ref-MISS-Bench, which covers 4 imaging modalities and 20 different organs and lesions. Experimental results on this benchmark demonstrate that TPP consistently outperforms state-of-the-art methods in both medical segmentation and referring video object segmentation.