Segmentation Ability Map: Interpret deep features for medical image segmentation

TL;DR

This paper introduces ProtoSeg, a method to interpret deep features in medical image segmentation by quantifying their segmentation abilities across layers, providing insights into neural network interpretability and performance estimation.

Contribution

The paper proposes a novel ProtoSeg method that measures deep feature segmentation abilities and offers a performance estimation without ground-truth, enhancing interpretability of CNNs in medical imaging.

Findings

SA scores quantify feature segmentation abilities across layers.

Mean SA score estimates model performance without ground-truth.

ProtoSeg provides insights into deep features for various medical images.

Abstract

Deep convolutional neural networks (CNNs) have been widely used for medical image segmentation. In most studies, only the output layer is exploited to compute the final segmentation results and the hidden representations of the deep learned features have not been well understood. In this paper, we propose a prototype segmentation (ProtoSeg) method to compute a binary segmentation map based on deep features. We measure the segmentation abilities of the features by computing the Dice between the feature segmentation map and ground-truth, named as the segmentation ability score (SA score for short). The corresponding SA score can quantify the segmentation abilities of deep features in different layers and units to understand the deep neural networks for segmentation. In addition, our method can provide a mean SA score which can give a performance estimation of the output on the test images without ground-truth. Finally, we use the proposed ProtoSeg method to compute the segmentation map directly on input images to further understand the segmentation ability of each input image. Results are presented on segmenting tumors in brain MRI, lesions in skin images, COVID-related abnormality in CT images, prostate segmentation in abdominal MRI, and pancreatic mass segmentation in CT images. Our method can provide new insights for interpreting and explainable AI systems for medical image segmentation. Our code is available on: \url{https://github.com/shengfly/ProtoSeg}.