# PromptSeg: An End-to-End Universal Medical Image Segmentation Method via Visual Prompts

## TL;DR

PromptSeg is a new AI method for medical image segmentation that improves generalization across different tasks using visual prompts.

## Contribution

PromptSeg introduces a universal framework for medical image segmentation using visual prompts and information-theoretic principles.

## Key findings

- PromptSeg outperforms state-of-the-art methods on CT and MRI datasets.
- The method shows strong generalization across multi-modality and unseen datasets.
- Only a few annotated visual prompts are needed for new tasks without retraining.

## Abstract

Deep learning has achieved remarkable advancements in medical image segmentation, yet its generalization capability across unseen tasks remains a significant challenge. The variety of task objectives, disease-dependent labeling variations, and multi-center data contribute to the high uncertainty of task-specific models on unseen distributions. In this study, we propose PromptSeg, an innovative Transformer-based unified framework for universal 2D medical image segmentation. From an information-theoretic perspective, PromptSeg formulates the segmentation process as a conditional entropy minimization problem, utilizing visual prompts as side information to reduce the uncertainty of the target task. Guided by the information bottleneck principle, PromptSeg aims to utilize the provided visual prompts to filter out redundant noise and learn contextual representations, thereby breaking the restrictions of the task-specific paradigm. When faced with unseen datasets or segmentation targets, our method only requires a few annotated visual prompt pairs to extract task-specific semantics and segment the query images without retraining. Extensive experiments on CT and MRI datasets demonstrate that PromptSeg not only outperforms state-of-the-art methods but also exhibits strong multi-modality generalization capabilities.