What Can We Learn from Inter-Annotator Variability in Skin Lesion Segmentation?

TL;DR

This study investigates inter-annotator variability in skin lesion segmentation, revealing its correlation with malignancy and demonstrating how modeling this variability improves segmentation accuracy using a new multi-annotator dataset.

Contribution

We introduce IMA++, the largest multi-annotator skin lesion segmentation dataset, and show how inter-annotator agreement relates to malignancy and can enhance model performance.

Findings

Inter-annotator agreement correlates with lesion malignancy (p<0.001).

Inter-annotator agreement can be predicted from dermoscopic images with MAE of 0.108.

Using inter-annotator agreement as a feature improves segmentation accuracy by 4.2%.

Abstract

Medical image segmentation exhibits intra- and inter-annotator variability due to ambiguous object boundaries, annotator preferences, expertise, and tools, among other factors. Lesions with ambiguous boundaries, e.g., spiculated or infiltrative nodules, or irregular borders per the ABCD rule, are particularly prone to disagreement and are often associated with malignancy. In this work, we curate IMA++, the largest multi-annotator skin lesion segmentation dataset, on which we conduct an in-depth study of variability due to annotator, malignancy, tool, and skill factors. We find a statistically significant (p<0.001) association between inter-annotator agreement (IAA), measured using Dice, and the malignancy of skin lesions. We further show that IAA can be accurately predicted directly from dermoscopic images, achieving a mean absolute error of 0.108. Finally, we leverage this association by utilizing IAA as a "soft" clinical feature within a multi-task learning objective, yielding a 4.2% improvement in balanced accuracy averaged across multiple model architectures and across IMA++ and four public dermoscopic datasets. The code is available at https://github.com/sfu-mial/skin-IAV.