Accuracy comparison across face recognition algorithms: Where are we on measuring race bias?

TL;DR

This paper examines the factors influencing race bias in face recognition algorithms, analyzing how data quality, scenario settings, and thresholds affect bias measurement across different algorithms and populations.

Contribution

It provides a comprehensive framework and checklist for assessing race bias in face recognition, considering data, scenario, and threshold factors.

Findings

Race bias increases with dataset difficulty.

Bias varies with identification thresholds.

Demographic constraints impact accuracy estimates.

Abstract

Previous generations of face recognition algorithms differ in accuracy for images of different races (race bias). Here, we present the possible underlying factors (data-driven and scenario modeling) and methodological considerations for assessing race bias in algorithms. We discuss data driven factors (e.g., image quality, image population statistics, and algorithm architecture), and scenario modeling factors that consider the role of the "user" of the algorithm (e.g., threshold decisions and demographic constraints). To illustrate how these issues apply, we present data from four face recognition algorithms (a previous-generation algorithm and three deep convolutional neural networks, DCNNs) for East Asian and Caucasian faces. First, dataset difficulty affected both overall recognition accuracy and race bias, such that race bias increased with item difficulty. Second, for all four algorithms, the degree of bias varied depending on the identification decision threshold. To achieve equal false accept rates (FARs), East Asian faces required higher identification thresholds than Caucasian faces, for all algorithms. Third, demographic constraints on the formulation of the distributions used in the test, impacted estimates of algorithm accuracy. We conclude that race bias needs to be measured for individual applications and we provide a checklist for measuring this bias in face recognition algorithms.