Colorimetric skin tone scale for improved accuracy and reduced perceptual bias of human skin tone annotations

TL;DR

This paper introduces a new colorimetric skin tone scale that improves accuracy and reduces perceptual bias in human skin tone annotations, addressing limitations of existing subjective rating scales.

Contribution

The study develops and validates a novel colorimetric skin tone scale based on measured skin color, enhancing annotation precision and reducing bias compared to traditional scales.

Findings

The CST scale is more sensitive and consistent than existing scales.

Manual ratings are biased by race and other factors.

Measured skin tone correlates better with actual skin color.

Abstract

Human image datasets used to develop and evaluate technology should represent the diversity of human phenotypes, including skin tone. Datasets that include skin tone information frequently rely on manual skin tone ratings based on the Fitzpatrick Skin Type (FST) or the Monk Skin Tone (MST) scales in lieu of the actual measured skin tone of the image dataset subjects. However, perceived skin tone is subject to known biases and skin tone appearance in digital images can vary substantially depending on the capture camera and environment, confounding manual ratings. Surprisingly, the relationship between skin-tone ratings and measured skin tone has not been explored. To close this research gap, we measured the relationship between skin tone ratings from existing scales (FST, MST) and skin tone values measured by a calibrated colorimeter. We also propose and assess a novel Colorimetric Skin Tone (CST) scale developed based on prior colorimetric measurements. Using experiments requiring humans to rate their own skin tone and the skin tone of subjects in images, we show that the new CST scale is more sensitive, consistent, and colorimetrically accurate. While skin tone ratings appeared to correct for some color variation across images, they introduced biases related to race and other factors. These biases must be considered before using manual skin-tone ratings in technology evaluations or for engineering decisions.