Deep Quality Estimation: Creating Surrogate Models for Human Quality Ratings

TL;DR

This paper develops surrogate models to estimate human segmentation quality ratings, achieving accuracy comparable to intra-rater reliability on complex glioma segmentation tasks with scarce expert data.

Contribution

It introduces a method to approximate human quality ratings using surrogate models trained on limited data, applicable to 3D medical image segmentation.

Findings

Models approximate segmentation quality within human intra-rater reliability margins.

Effective despite operating on 2D images and limited training data.

Potential for real-time quality assessment and dataset curation.

Abstract

Human ratings are abstract representations of segmentation quality. To approximate human quality ratings on scarce expert data, we train surrogate quality estimation models. We evaluate on a complex multi-class segmentation problem, specifically glioma segmentation, following the BraTS annotation protocol. The training data features quality ratings from 15 expert neuroradiologists on a scale ranging from 1 to 6 stars for various computer-generated and manual 3D annotations. Even though the networks operate on 2D images and with scarce training data, we can approximate segmentation quality within a margin of error comparable to human intra-rater reliability. Segmentation quality prediction has broad applications. While an understanding of segmentation quality is imperative for successful clinical translation of automatic segmentation quality algorithms, it can play an essential role in training new segmentation models. Due to the split-second inference times, it can be directly applied within a loss function or as a fully-automatic dataset curation mechanism in a federated learning setting.