Let's agree to disagree: learning highly debatable multirater labelling

TL;DR

This paper introduces a deep learning approach that models individual rater behaviors and consensus to improve detection of challenging brain lesions, addressing variability and disagreement among human experts.

Contribution

It presents a novel joint modeling framework for individual and consensus labels in multirater settings, enhancing detection accuracy and understanding rater consistency.

Findings

Joint modeling improves detection performance over direct consensus prediction.

The approach characterizes human rater consistency effectively.

Significant performance gains demonstrated in brain lesion detection.

Abstract

Classification and differentiation of small pathological objects may greatly vary among human raters due to differences in training, expertise and their consistency over time. In a radiological setting, objects commonly have high within-class appearance variability whilst sharing certain characteristics across different classes, making their distinction even more difficult. As an example, markers of cerebral small vessel disease, such as enlarged perivascular spaces (EPVS) and lacunes, can be very varied in their appearance while exhibiting high inter-class similarity, making this task highly challenging for human raters. In this work, we investigate joint models of individual rater behaviour and multirater consensus in a deep learning setting, and apply it to a brain lesion object-detection task. Results show that jointly modelling both individual and consensus estimates leads to significant improvements in performance when compared to directly predicting consensus labels, while also allowing the characterization of human-rater consistency.