3SGAN: Semi-Supervised and Multi-Task GAN for Stain Normalization and Nuclei Segmentation of Histopathological Images

TL;DR

3SGAN is an AI system that standardizes staining colors and identifies cell nuclei in cancer images using minimal labeled data, improving cancer diagnosis efficiency.

Contribution

3SGAN introduces a semi-supervised, multi-task GAN framework that simultaneously performs stain normalization and nucleus segmentation with only 5% labeled data.

Findings

3SGAN achieved an F1-score of 0.8140 and mean IoU of 0.8201 in nucleus segmentation.

It demonstrated strong generalizability across diverse staining protocols and tissue types.

Abstract

Analyzing cancer cells from microscope images is challenging due to inconsistent staining colors and the time-consuming nature of manually labeling cells. We developed 3SGAN, an artificial intelligence system that simultaneously standardizes staining colors and identifies cell nuclei using only 5% manually labeled data. Demonstrating superior performance on a large dataset of 1408 Whole-Slide Images (WSIs) from two medical institutions—encompassing 101 distinct staining styles—our method significantly outperformed existing approaches in both tasks. This technology could accelerate cancer diagnosis and reduce the workload of pathologists while maintaining high accuracy with minimal labeled data.

Background/Objectives: Variations in staining styles—arising from differences in tissue preparation, scanners, and laboratory protocols—severely compromise the robustness of automated cell segmentation algorithms in digital pathology. Moreover, manual nucleus annotation is extremely labor-intensive, leading to a scarcity of large-scale, fully annotated datasets for supervised nucleus segmentation. This study proposes a novel framework that simultaneously mitigates staining variability and achieves high-accuracy nucleus segmentation using only minimal annotations. Methods: We present 3SGAN, a multi-task dual-branch generative adversarial network (GAN) that jointly performs stain normalization and nucleus segmentation in a semi-supervised manner. The framework adopts a teacher–student paradigm: a lightweight teacher model (AttCycle) equipped with attention gates generates reliable pseudo-labels, while a high-capacity student model (TransCycle) leveraging a hybrid CNN–Transformer architecture further refines performance. 3SGAN was trained and evaluated on a large dataset of 1408 Whole-Slide Images (WSIs) from two medical institutions, encompassing 101 distinct staining styles, with nucleus-level annotations required for only 5% of the data. Results: 3SGAN significantly outperformed state-of-the-art methods, achieving superior segmentation accuracy with an F1-score of 0.8140, mean IoU of 0.8201, and AJI of 0.6915. Simultaneously, it demonstrated substantial improvements in stain normalization quality, yielding a low RMSE of 0.0908, high PSNR of 21.0615, and SSIM of 0.8556 on the internal test set. External validation on independent MoNuSeg and PanNuke datasets, as well as on previously untested tumor-rich non-ROI regions from our in-house WSIs, confirmed strong generalizability with excellent stain normalization and top-tier segmentation accuracy across diverse staining protocols, tissue types, and pathological patterns. Conclusions: The proposed 3SGAN framework demonstrates that high-performance nucleus segmentation and stain normalization can be achieved with minimal annotation requirements, offering a practical and scalable solution for digital pathology applications across diverse clinical settings and staining protocols.

Figures