Impact of individual rater style on deep learning uncertainty in medical imaging segmentation

TL;DR

This study investigates how individual rater styles, including bias and consistency, influence deep learning uncertainty in medical image segmentation, revealing significant correlations and the importance of multi-center label fusion.

Contribution

It quantifies rater style effects on model uncertainty and demonstrates that multi-center consensuses better mitigate rater-specific biases in medical segmentation.

Findings

Rater bias correlates with deep learning uncertainty (R^2=0.60 and 0.93).

Multi-center consensuses reduce uncertainty more effectively than single-center ones.

Rater style is predominantly center-specific, affecting model training and uncertainty.

Abstract

While multiple studies have explored the relation between inter-rater variability and deep learning model uncertainty in medical segmentation tasks, little is known about the impact of individual rater style. This study quantifies rater style in the form of bias and consistency and explores their impacts when used to train deep learning models. Two multi-rater public datasets were used, consisting of brain multiple sclerosis lesion and spinal cord grey matter segmentation. On both datasets, results show a correlation ($R^2 = 0.60$ and $0.93$) between rater bias and deep learning uncertainty. The impact of label fusion between raters' annotations on this relationship is also explored, and we show that multi-center consensuses are more effective than single-center consensuses to reduce uncertainty, since rater style is mostly center-specific.