On The Robustness of Foundational 3D Medical Image Segmentation Models Against Imprecise Visual Prompts

TL;DR

This paper investigates how robust 3D medical image segmentation models are when faced with imprecise visual prompts, revealing their reliance on shape and spatial cues and their resilience to various perturbations.

Contribution

It systematically studies the impact of prompt imprecision on foundational 3D medical segmentation models, highlighting their vulnerabilities and robustness.

Findings

Models rely heavily on shape and spatial cues.

Certain perturbations significantly degrade segmentation performance.

Some models show resilience to specific types of prompt imprecision.

Abstract

While 3D foundational models have shown promise for promptable segmentation of medical volumes, their robustness to imprecise prompts remains under-explored. In this work, we aim to address this gap by systematically studying the effect of various controlled perturbations of dense visual prompts, that closely mimic real-world imprecision. By conducting experiments with two recent foundational models on a multi-organ abdominal segmentation task, we reveal several facets of promptable medical segmentation, especially pertaining to reliance on visual shape and spatial cues, and the extent of resilience of models towards certain perturbations. Codes are available at: https://github.com/ucsdbiag/Prompt-Robustness-MedSegFMs