Revisiting Skin Tone Fairness in Dermatological Lesion Classification

TL;DR

This paper critically examines ITA-based skin tone classification methods in dermatological datasets, highlighting their inconsistencies and emphasizing the need for diverse, well-annotated datasets to improve fairness assessments in AI skin lesion classification.

Contribution

The study compares four ITA-based skin tone classification approaches, identifies their disagreements, and discusses dataset limitations affecting fairness analysis in dermatology AI.

Findings

High disagreement among ITA-based skin tone methods

Limited dataset diversity hampers fairness analysis

Recommendations for robust skin tone estimation and dataset collection

Abstract

Addressing fairness in lesion classification from dermatological images is crucial due to variations in how skin diseases manifest across skin tones. However, the absence of skin tone labels in public datasets hinders building a fair classifier. To date, such skin tone labels have been estimated prior to fairness analysis in independent studies using the Individual Typology Angle (ITA). Briefly, ITA calculates an angle based on pixels extracted from skin images taking into account the lightness and yellow-blue tints. These angles are then categorised into skin tones that are subsequently used to analyse fairness in skin cancer classification. In this work, we review and compare four ITA-based approaches of skin tone classification on the ISIC18 dataset, a common benchmark for assessing skin cancer classification fairness in the literature. Our analyses reveal a high disagreement among previously published studies demonstrating the risks of ITA-based skin tone estimation methods. Moreover, we investigate the causes of such large discrepancy among these approaches and find that the lack of diversity in the ISIC18 dataset limits its use as a testbed for fairness analysis. Finally, we recommend further research on robust ITA estimation and diverse dataset acquisition with skin tone annotation to facilitate conclusive fairness assessments of artificial intelligence tools in dermatology. Our code is available at https://github.com/tkalbl/RevisitingSkinToneFairness.