Characterizing the Variability in Face Recognition Accuracy Relative to Race

TL;DR

This study systematically investigates racial disparities in face recognition accuracy, revealing significant differences in error rates between African-American and Caucasian cohorts across multiple matchers and emphasizing the importance of fixed thresholds and image quality.

Contribution

It provides a detailed analysis of accuracy differences between racial groups, highlighting the impact of image quality and threshold choices on face recognition fairness.

Findings

African-American cohort has higher false match rates at fixed thresholds

ICAO-compliant images improve recognition accuracy across cohorts

ROC curves may not reflect operational performance differences

Abstract

Many recent news headlines have labeled face recognition technology as biased or racist. We report on a methodical investigation into differences in face recognition accuracy between African-American and Caucasian image cohorts of the MORPH dataset. We find that, for all four matchers considered, the impostor and the genuine distributions are statistically significantly different between cohorts. For a fixed decision threshold, the African-American image cohort has a higher false match rate and a lower false non-match rate. ROC curves compare verification rates at the same false match rate, but the different cohorts achieve the same false match rate at different thresholds. This means that ROC comparisons are not relevant to operational scenarios that use a fixed decision threshold. We show that, for the ResNet matcher, the two cohorts have approximately equal separation of impostor and genuine distributions. Using ICAO compliance as a standard of image quality, we find that the initial image cohorts have unequal rates of good quality images. The ICAO-compliant subsets of the original image cohorts show improved accuracy, with the main effect being to reducing the low-similarity tail of the genuine distributions.