ENHANCE (ENriching Health data by ANnotations of Crowd and Experts): A case study for skin lesion classification

TL;DR

ENHANCE introduces a new annotated skin lesion dataset combining expert and non-expert labels, and demonstrates that multi-task learning with these annotations can enhance CNN performance for skin lesion classification.

Contribution

The paper provides a novel multi-annotated skin lesion dataset and shows how non-expert annotations can improve CNN models through multi-task learning.

Findings

Weak correlation between non-expert annotations and diagnostic labels

Low agreement among different annotation sources

Multi-task learning with annotations improves CNN accuracy

Abstract

We present ENHANCE, an open dataset with multiple annotations to complement the existing ISIC and PH2 skin lesion classification datasets. This dataset contains annotations of visual ABC (asymmetry, border, colour) features from non-expert annotation sources: undergraduate students, crowd workers from Amazon MTurk and classic image processing algorithms. In this paper we first analyse the correlations between the annotations and the diagnostic label of the lesion, as well as study the agreement between different annotation sources. Overall we find weak correlations of non-expert annotations with the diagnostic label, and low agreement between different annotation sources. We then study multi-task learning (MTL) with the annotations as additional labels, and show that non-expert annotations can improve (ensembles of) state-of-the-art convolutional neural networks via MTL. We hope that our dataset can be used in further research into multiple annotations and/or MTL. All data and models are available on Github: https://github.com/raumannsr/ENHANCE.