Rule-based outlier detection of AI-generated anatomy segmentations

TL;DR

This paper introduces a rule-based heuristic approach to evaluate the quality of AI-generated anatomical segmentations in medical imaging, addressing the lack of expert annotations and accuracy assessments in large datasets.

Contribution

The paper develops heuristics for assessing segmentation quality, enabling consistency checks and accuracy estimation without requiring expert annotations.

Findings

Heuristics effectively identify outlier segmentations.

Method improves reliability of AI-generated annotations.

Code and tools are publicly available for community use.

Abstract

There is a dire need for medical imaging datasets with accompanying annotations to perform downstream patient analysis. However, it is difficult to manually generate these annotations, due to the time-consuming nature, and the variability in clinical conventions. Artificial intelligence has been adopted in the field as a potential method to annotate these large datasets, however, a lack of expert annotations or ground truth can inhibit the adoption of these annotations. We recently made a dataset publicly available including annotations and extracted features of up to 104 organs for the National Lung Screening Trial using the TotalSegmentator method. However, the released dataset does not include expert-derived annotations or an assessment of the accuracy of the segmentations, limiting its usefulness. We propose the development of heuristics to assess the quality of the segmentations, providing methods to measure the consistency of the annotations and a comparison of results to the literature. We make our code and related materials publicly available at https://github.com/ImagingDataCommons/CloudSegmentatorResults and interactive tools at https://huggingface.co/spaces/ImagingDataCommons/CloudSegmentatorResults.