TrustSkin: A Fairness Pipeline for Trustworthy Facial Affect Analysis Across Skin Tone

TL;DR

This paper evaluates skin tone measurement methods for facial affect analysis fairness, revealing that perceptual measures like $H^*$-$L^*$ improve subgroup fairness diagnostics and highlight disparities affecting darker skin tones.

Contribution

It introduces a modular fairness pipeline using perceptual skin tone estimation, improving fairness assessment and diagnosis in facial affect analysis systems.

Findings

Dark skin tones are severely underrepresented (~2%) in datasets.

ITA method is sensitive to lighting, affecting fairness evaluation.

Perceptual $H^*$-$L^*$ method provides more consistent subgrouping and clearer fairness diagnostics.

Abstract

Understanding how facial affect analysis (FAA) systems perform across different demographic groups requires reliable measurement of sensitive attributes such as ancestry, often approximated by skin tone, which itself is highly influenced by lighting conditions. This study compares two objective skin tone classification methods: the widely used Individual Typology Angle (ITA) and a perceptually grounded alternative based on Lightness ($L^*$) and Hue ($H^*$). Using AffectNet and a MobileNet-based model, we assess fairness across skin tone groups defined by each method. Results reveal a severe underrepresentation of dark skin tones ($\sim 2 \%$), alongside fairness disparities in F1-score (up to 0.08) and TPR (up to 0.11) across groups. While ITA shows limitations due to its sensitivity to lighting, the $H^*$-$L^*$ method yields more consistent subgrouping and enables clearer diagnostics through metrics such as Equal Opportunity. Grad-CAM analysis further highlights differences in model attention patterns by skin tone, suggesting variation in feature encoding. To support future mitigation efforts, we also propose a modular fairness-aware pipeline that integrates perceptual skin tone estimation, model interpretability, and fairness evaluation. These findings emphasize the relevance of skin tone measurement choices in fairness assessment and suggest that ITA-based evaluations may overlook disparities affecting darker-skinned individuals.