Reliability and Validity of Image-Based and Self-Reported Skin Phenotype Metrics

TL;DR

This study evaluates the reliability of image-based and self-reported skin-tone metrics, revealing that controlled, objective measures are essential for accurate biometric performance assessments across demographic groups.

Contribution

It demonstrates the unreliability of uncontrolled image-based and self-reported skin-tone measures and advocates for objective, controlled methods in biometric evaluations.

Findings

Image-based FALMs vary significantly across images of the same individual.

Fitzpatrick Skin Types poorly predict actual skin-tone.

Noisy FALM estimates lead to incorrect demographic differential analysis.

Abstract

With increasing adoption of face recognition systems, it is important to ensure adequate performance of these technologies across demographic groups. Recently, phenotypes such as skin-tone, have been proposed as superior alternatives to traditional race categories when exploring performance differentials. However, there is little consensus regarding how to appropriately measure skin-tone in evaluations of biometric performance or in AI more broadly. In this study, we explore the relationship between face-area-lightness-measures (FALMs) estimated from images and ground-truth skin readings collected using a device designed to measure human skin. FALMs estimated from different images of the same individual varied significantly relative to ground-truth FALM. This variation was only reduced by greater control of acquisition (camera, background, and environment). Next, we compare ground-truth FALM to Fitzpatrick Skin Types (FST) categories obtained using the standard, in-person, medical survey and show FST is poorly predictive of skin-tone. Finally, we show how noisy estimation of FALM leads to errors selecting explanatory factors for demographic differentials. These results demonstrate that measures of skin-tone for biometric performance evaluations must come from objective, characterized, and controlled sources. Further, despite this being a currently practiced approach, estimating FST categories and FALMs from uncontrolled imagery does not provide an appropriate measure of skin-tone.