Evaluating Machine Learning-based Skin Cancer Diagnosis

TL;DR

This study assesses the reliability, explainability, and fairness of deep learning models for skin cancer detection using dermatoscopic images, highlighting strengths and disparities across skin tones and lesion types.

Contribution

It provides a comprehensive evaluation of two CNN models' performance, explainability, and fairness, and introduces a postprocessing method to improve fairness across skin tones.

Findings

Both models highlight relevant features for most lesion types.

Models show fairness across sex but disparities across skin tones.

Postprocessing improves fairness by reducing false negative rate differences.

Abstract

This study evaluates the reliability of two deep learning models for skin cancer detection, focusing on their explainability and fairness. Using the HAM10000 dataset of dermatoscopic images, the research assesses two convolutional neural network architectures: a MobileNet-based model and a custom CNN model. Both models are evaluated for their ability to classify skin lesions into seven categories and to distinguish between dangerous and benign lesions. Explainability is assessed using Saliency Maps and Integrated Gradients, with results interpreted by a dermatologist. The study finds that both models generally highlight relevant features for most lesion types, although they struggle with certain classes like seborrheic keratoses and vascular lesions. Fairness is evaluated using the Equalized Odds metric across sex and skin tone groups. While both models demonstrate fairness across sex groups, they show significant disparities in false positive and false negative rates between light and dark skin tones. A Calibrated Equalized Odds postprocessing strategy is applied to mitigate these disparities, resulting in improved fairness, particularly in reducing false negative rate differences. The study concludes that while the models show promise in explainability, further development is needed to ensure fairness across different skin tones. These findings underscore the importance of rigorous evaluation of AI models in medical applications, particularly in diverse population groups.