DualProtoSeg: Simple and Efficient Design with Text- and Image-Guided Prototype Learning for Weakly Supervised Histopathology Image Segmentation

TL;DR

DualProtoSeg introduces a prototype-driven framework combining text and image cues, along with multi-scale features, to enhance weakly supervised histopathology image segmentation, surpassing state-of-the-art methods.

Contribution

It presents a novel dual-modal prototype bank with text and image prototypes, and incorporates multi-scale pyramid modules for improved localization in weakly supervised segmentation.

Findings

Outperforms existing state-of-the-art on BCSS-WSSS benchmark.

Text description diversity and context length improve segmentation quality.

Combining textual and visual prototypes enhances region discovery.

Abstract

Weakly supervised semantic segmentation (WSSS) in histopathology seeks to reduce annotation cost by learning from image-level labels, yet it remains limited by inter-class homogeneity, intra-class heterogeneity, and the region-shrinkage effect of CAM-based supervision. We propose a simple and effective prototype-driven framework that leverages vision-language alignment to improve region discovery under weak supervision. Our method integrates CoOp-style learnable prompt tuning to generate text-based prototypes and combines them with learnable image prototypes, forming a dual-modal prototype bank that captures both semantic and appearance cues. To address oversmoothing in ViT representations, we incorporate a multi-scale pyramid module that enhances spatial precision and improves localization quality. Experiments on the BCSS-WSSS benchmark show that our approach surpasses existing state-of-the-art methods, and detailed analyses demonstrate the benefits of text description diversity, context length, and the complementary behavior of text and image prototypes. These results highlight the effectiveness of jointly leveraging textual semantics and visual prototype learning for WSSS in digital pathology.