Refining 3D Medical Segmentation with Verbal Instruction

TL;DR

This paper introduces CoWTalk, a benchmark for 3D arterial shape correction with verbal instructions, and proposes an iterative refinement model that improves shape predictions using natural language feedback.

Contribution

The paper presents a new benchmark dataset and a novel language-interactive refinement method for 3D medical shape segmentation.

Findings

Significant improvement over corrupted inputs

Competitive performance against baseline methods

Feasibility of language-driven shape refinement

Abstract

Accurate 3D anatomical segmentation is essential for clinical diagnosis and surgical planning. However, automated models frequently generate suboptimal shape predictions due to factors such as limited and imbalanced training data, inadequate labeling quality, and distribution shifts between training and deployment settings. A natural solution is to iteratively refine the predicted shape based on the radiologists' verbal instructions. However, this is hindered by the scarcity of paired data that explicitly links erroneous shapes to corresponding corrective instructions. As an initial step toward addressing this limitation, we introduce CoWTalk, a benchmark comprising 3D arterial anatomies with controllable synthesized anatomical errors and their corresponding repairing instructions. Building on this benchmark, we further propose an iterative refinement model that represents 3D shapes as vector sets and interacts with textual instructions to progressively update the target shape. Experimental results demonstrate that our method achieves significant improvements over corrupted inputs and competitive baselines, highlighting the feasibility of language-driven clinician-in-the-loop refinement for 3D medical shapes modeling.