Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)

TL;DR

This paper reports on the 2018 ISIC skin lesion analysis challenge, highlighting the largest dermoscopic image dataset, novel evaluation protocols, and insights into algorithm performance and generalization in melanoma detection.

Contribution

It introduces a large-scale public skin image dataset, new evaluation protocols, and benchmarks for algorithm performance and generalization in melanoma detection.

Findings

Top segmentation algorithms fail on over 10% of images

Algorithms with similar test performance can differ in generalization ability

The challenge sets new standards for evaluating healthcare machine learning tools

Abstract

This work summarizes the results of the largest skin image analysis challenge in the world, hosted by the International Skin Imaging Collaboration (ISIC), a global partnership that has organized the world's largest public repository of dermoscopic images of skin. The challenge was hosted in 2018 at the Medical Image Computing and Computer Assisted Intervention (MICCAI) conference in Granada, Spain. The dataset included over 12,500 images across 3 tasks. 900 users registered for data download, 115 submitted to the lesion segmentation task, 25 submitted to the lesion attribute detection task, and 159 submitted to the disease classification task. Novel evaluation protocols were established, including a new test for segmentation algorithm performance, and a test for algorithm ability to generalize. Results show that top segmentation algorithms still fail on over 10% of images on average, and algorithms with equal performance on test data can have different abilities to generalize. This is an important consideration for agencies regulating the growing set of machine learning tools in the healthcare domain, and sets a new standard for future public challenges in healthcare.