CellGenNet: A Knowledge-Distilled Framework for Robust Cell Segmentation in Cancer Tissues

TL;DR

CellGenNet introduces a knowledge-distilled framework utilizing a student-teacher model with hybrid loss and regularization techniques to enhance robust nuclei segmentation in diverse cancer tissue images with limited supervision.

Contribution

The paper presents a novel knowledge distillation approach with a hybrid loss and regularization strategies for improved cell segmentation in histopathology images.

Findings

Enhanced segmentation accuracy over baseline methods

Improved generalization across diverse tissue types

Effective handling of class imbalance in nuclei segmentation

Abstract

Accurate nuclei segmentation in microscopy whole slide images (WSIs) remains challenging due to variability in staining, imaging conditions, and tissue morphology. We propose CellGenNet, a knowledge distillation framework for robust cross-tissue cell segmentation under limited supervision. CellGenNet adopts a student-teacher architecture, where a capacity teacher is trained on sparse annotations and generates soft pseudo-labels for unlabeled regions. The student is optimized using a joint objective that integrates ground-truth labels, teacher-derived probabilistic targets, and a hybrid loss function combining binary cross-entropy and Tversky loss, enabling asymmetric penalties to mitigate class imbalance and better preserve minority nuclear structures. Consistency regularization and layerwise dropout further stabilize feature representations and promote reliable feature transfer. Experiments across diverse cancer tissue WSIs show that CellGenNet improves segmentation accuracy and generalization over supervised and semi-supervised baselines, supporting scalable and reproducible histopathology analysis.