PRISM: A Promptable and Robust Interactive Segmentation Model with Visual Prompts

TL;DR

PRISM is a novel interactive segmentation model for 3D medical images that uses visual prompts and iterative, confidence, and corrective learning principles to achieve near-human segmentation accuracy.

Contribution

PRISM introduces a robust, promptable segmentation framework with hybrid encoders and iterative learning, advancing medical image segmentation accuracy.

Findings

Achieves near-human segmentation performance on multiple datasets.

Significantly outperforms existing prompt-based and non-prompt segmentation methods.

Demonstrates robustness across diverse anatomical variations and ambiguous boundaries.

Abstract

In this paper, we present PRISM, a Promptable and Robust Interactive Segmentation Model, aiming for precise segmentation of 3D medical images. PRISM accepts various visual inputs, including points, boxes, and scribbles as sparse prompts, as well as masks as dense prompts. Specifically, PRISM is designed with four principles to achieve robustness: (1) Iterative learning. The model produces segmentations by using visual prompts from previous iterations to achieve progressive improvement. (2) Confidence learning. PRISM employs multiple segmentation heads per input image, each generating a continuous map and a confidence score to optimize predictions. (3) Corrective learning. Following each segmentation iteration, PRISM employs a shallow corrective refinement network to reassign mislabeled voxels. (4) Hybrid design. PRISM integrates hybrid encoders to better capture both the local and global information. Comprehensive validation of PRISM is conducted using four public datasets for tumor segmentation in the colon, pancreas, liver, and kidney, highlighting challenges caused by anatomical variations and ambiguous boundaries in accurate tumor identification. Compared to state-of-the-art methods, both with and without prompt engineering, PRISM significantly improves performance, achieving results that are close to human levels. The code is publicly available at https://github.com/MedICL-VU/PRISM.