Comparative analysis of deep learning approaches for AgNOR-stained cytology samples interpretation

TL;DR

This paper compares deep learning models for analyzing AgNOR-stained cytology slides, highlighting the effectiveness of semantic segmentation with U-Net and instance segmentation with Mask R-CNN for nucleus and NOR detection.

Contribution

It provides a comparative analysis of deep learning approaches for AgNOR slide interpretation, proposing a cascade model combining semantic and instance segmentation.

Findings

Semantic segmentation with U-Net achieves IoU of 0.83 for nuclei.

Mask R-CNN with ResNet-50 achieves IoU of 0.61 for nuclei.

Combined models can improve accuracy in nucleus and NOR detection.

Abstract

Cervical cancer is a public health problem, where the treatment has a better chance of success if detected early. The analysis is a manual process which is subject to a human error, so this paper provides a way to analyze argyrophilic nucleolar organizer regions (AgNOR) stained slide using deep learning approaches. Also, this paper compares models for instance and semantic detection approaches. Our results show that the semantic segmentation using U-Net with ResNet-18 or ResNet-34 as the backbone have similar results, and the best model shows an IoU for nucleus, cluster, and satellites of 0.83, 0.92, and 0.99 respectively. For instance segmentation, the Mask R-CNN using ResNet-50 performs better in the visual inspection and has a 0.61 of the IoU metric. We conclude that the instance segmentation and semantic segmentation models can be used in combination to make a cascade model able to select a nucleus and subsequently segment the nucleus and its respective nucleolar organizer regions (NORs).