Large-Scale Dataset and Benchmark for Skin Tone Classification in the Wild

TL;DR

This paper introduces a large, annotated skin tone dataset and benchmarks various models, demonstrating deep learning's effectiveness and enabling fairness auditing in skin tone classification.

Contribution

The work provides a comprehensive skin tone dataset, benchmarks existing methods, and proposes SkinToneNet for improved fairness assessment and generalization.

Findings

Deep learning models outperform classic methods in skin tone classification.

The dataset enables more accurate fairness analysis across diverse populations.

SkinToneNet achieves state-of-the-art generalization on out-of-domain data.

Abstract

Deep learning models often inherit biases from their training data. While fairness across gender and ethnicity is well-studied, fine-grained skin tone analysis remains a challenge due to the lack of granular, annotated datasets. Existing methods often rely on the medical 6-tone Fitzpatrick scale, which lacks visual representativeness, or use small, private datasets that prevent reproducibility, or often rely on classic computer vision pipelines, with a few using deep learning. They overlook issues like train-test leakage and dataset imbalance, and are limited by small or unavailable datasets. In this work, we present a comprehensive framework for skin tone fairness. First, we introduce the STW, a large-scale, open-access dataset comprising 42,313 images from 3,564 individuals, labeled using the 10-tone MST scale. Second, we benchmark both Classic Computer Vision (SkinToneCCV) and Deep Learning approaches, demonstrating that classic models provide near-random results, while deep learning reaches nearly annotator accuracy. Finally, we propose SkinToneNet, a fine-tuned ViT that achieves state-of-the-art generalization on out-of-domain data, which enables reliable fairness auditing of public datasets like CelebA and VGGFace2. This work provides state-of-the-art results in skin tone classification and fairness assessment. Code and data available soon