Human Perception-based Evaluation Criterion for Ultra-high Resolution Cell Membrane Segmentation

TL;DR

This paper introduces a new high-resolution cell membrane dataset and proposes a human perception-based evaluation criterion, PHD, to better assess segmentation quality aligning with human judgment.

Contribution

The paper provides the largest annotated electron microscopy dataset for cell membranes and introduces PHD, a novel evaluation metric based on human perception.

Findings

Current evaluation metrics are inconsistent with human perception.

PHD correlates better with human subjective judgment.

New dataset U-RISC enhances research in cell membrane segmentation.

Abstract

Computer vision technology is widely used in biological and medical data analysis and understanding. However, there are still two major bottlenecks in the field of cell membrane segmentation, which seriously hinder further research: lack of sufficient high-quality data and lack of suitable evaluation criteria. In order to solve these two problems, this paper first proposes an Ultra-high Resolution Image Segmentation dataset for the Cell membrane, called U-RISC, the largest annotated Electron Microscopy (EM) dataset for the Cell membrane with multiple iterative annotations and uncompressed high-resolution raw data. During the analysis process of the U-RISC, we found that the current popular segmentation evaluation criteria are inconsistent with human perception. This interesting phenomenon is confirmed by a subjective experiment involving twenty people. Furthermore, to resolve this inconsistency, we propose a new evaluation criterion called Perceptual Hausdorff Distance (PHD) to measure the quality of cell membrane segmentation results. Detailed performance comparison and discussion of classic segmentation methods along with two iterative manual annotation results under existing evaluation criteria and PHD is given.