Crowd disagreement about medical images is informative

TL;DR

This paper investigates how disagreement among crowd annotators about medical images, specifically skin lesions, can provide valuable information beyond consensus labels, improving understanding and potentially aiding diagnosis.

Contribution

It demonstrates that disagreement measures, such as standard deviation, contain informative signals in medical image annotation, challenging the common practice of ignoring annotator disagreement.

Findings

Disagreement measures are informative for medical image analysis.

Mean annotations perform best for classification.

Crowd annotations dataset is publicly available.

Abstract

Classifiers for medical image analysis are often trained with a single consensus label, based on combining labels given by experts or crowds. However, disagreement between annotators may be informative, and thus removing it may not be the best strategy. As a proof of concept, we predict whether a skin lesion from the ISIC 2017 dataset is a melanoma or not, based on crowd annotations of visual characteristics of that lesion. We compare using the mean annotations, illustrating consensus, to standard deviations and other distribution moments, illustrating disagreement. We show that the mean annotations perform best, but that the disagreement measures are still informative. We also make the crowd annotations used in this paper available at \url{https://figshare.com/s/5cbbce14647b66286544}.