Segment Anything in Pathology Images with Natural Language

TL;DR

This paper introduces PathSegmentor, a novel text-prompted segmentation model for pathology images, supported by the extensive PathSeg dataset, enabling accurate, flexible, and interpretable segmentation for clinical pathology applications.

Contribution

The paper presents the first text-prompted foundation model for pathology image segmentation and introduces the comprehensive PathSeg dataset, advancing clinical applicability and interpretability.

Findings

PathSegmentor outperforms specialized models in accuracy.

It surpasses existing prompt-based models significantly.

The model generalizes well to external datasets.

Abstract

Pathology image segmentation is crucial in computational pathology for analyzing histological features relevant to cancer diagnosis and prognosis. However, current methods face major challenges in clinical applications due to limited annotated data and restricted category definitions. To address these limitations, we propose PathSegmentor, the first text-prompted segmentation foundation model designed specifically for pathology images. We also introduce PathSeg, the largest and most comprehensive dataset for pathology segmentation, built from 21 public sources and containing 275k image-mask-label triples across 160 diverse categories. With PathSegmentor, users can perform semantic segmentation using natural language prompts, eliminating the need for laborious spatial inputs such as points or boxes. Extensive experiments demonstrate that PathSegmentor outperforms specialized models with higher accuracy and broader applicability, while maintaining a compact architecture. It significantly surpasses existing spatial- and text-prompted models by 0.145 and 0.429 in overall Dice scores, respectively, showing strong robustness in segmenting complex structures and generalizing to external datasets. Moreover, PathSegmentor's outputs enhance the interpretability of diagnostic models through feature importance estimation and imaging biomarker discovery, offering pathologists evidence-based support for clinical decision-making. This work advances the development of explainable AI in precision oncology.