Progressive Vision-Language Prompt for Multi-Organ Multi-Class Cell Semantic Segmentation with Single Branch

TL;DR

This paper introduces MONCH, a novel single-branch vision-language model for multi-organ, multi-class cell segmentation that effectively captures hierarchical features and multimodal information, outperforming existing methods on challenging datasets.

Contribution

The paper proposes a hierarchical feature extraction and progressive prompt decoder in a single-branch framework, integrating vision-language input for improved multi-class cell segmentation.

Findings

MONCH outperforms state-of-the-art methods on PanNuke dataset.

Hierarchical features improve segmentation accuracy across cell sizes and shapes.

Multimodal integration enhances context understanding in complex tissue images.

Abstract

Pathological cell semantic segmentation is a fundamental technology in computational pathology, essential for applications like cancer diagnosis and effective treatment. Given that multiple cell types exist across various organs, with subtle differences in cell size and shape, multi-organ, multi-class cell segmentation is particularly challenging. Most existing methods employ multi-branch frameworks to enhance feature extraction, but often result in complex architectures. Moreover, reliance on visual information limits performance in multi-class analysis due to intricate textural details. To address these challenges, we propose a Multi-OrgaN multi-Class cell semantic segmentation method with a single brancH (MONCH) that leverages vision-language input. Specifically, we design a hierarchical feature extraction mechanism to provide coarse-to-fine-grained features for segmenting cells of various shapes, including high-frequency, convolutional, and topological features. Inspired by the synergy of textual and multi-grained visual features, we introduce a progressive prompt decoder to harmonize multimodal information, integrating features from fine to coarse granularity for better context capture. Extensive experiments on the PanNuke dataset, which has significant class imbalance and subtle cell size and shape variations, demonstrate that MONCH outperforms state-of-the-art cell segmentation methods and vision-language models. Codes and implementations will be made publicly available.