Prompt Group-Aware Training for Robust Text-Guided Nuclei Segmentation

TL;DR

This paper introduces a prompt group-aware training framework that enhances the robustness and consistency of text-guided nuclei segmentation models, reducing sensitivity to prompt variations without changing the model architecture.

Contribution

The proposed method reformulates prompt sensitivity as a group-wise consistency problem and introduces a training framework that improves robustness and generalization in text-guided segmentation.

Findings

Significant reduction in performance variance across prompt quality levels.

Average Dice score improvement of 2.16 points on six zero-shot cross-dataset tasks.

Consistent gains in segmentation robustness demonstrated across multiple datasets.

Abstract

Foundation models such as Segment Anything Model 3 (SAM3) enable flexible text-guided medical image segmentation, yet their predictions remain highly sensitive to prompt formulation. Even semantically equivalent descriptions can yield inconsistent masks, limiting reliability in clinical and pathology workflows. We reformulate prompt sensitivity as a group-wise consistency problem. Semantically related prompts are organized into \emph{prompt groups} sharing the same ground-truth mask, and a prompt group-aware training framework is introduced for robust text-guided nuclei segmentation. The approach combines (i) a quality-guided group regularization that leverages segmentation loss as an implicit ranking signal, and (ii) a logit-level consistency constraint with a stop-gradient strategy to align predictions within each group. The method requires no architectural modification and leaves inference unchanged. Extensive experiments on multi-dataset nuclei benchmarks show consistent gains under textual prompting and markedly reduced performance variance across prompt quality levels. On six zero-shot cross-dataset tasks, our method improves Dice by an average of 2.16 points. These results demonstrate improved robustness and generalization for vision-language segmentation in computational pathology.