Statistical Methods for Assessing Differences in False Non-Match Rates Across Demographic Groups

TL;DR

This paper introduces two statistical methods, including a bootstrap hypothesis test, to evaluate whether differences in false non-match rates across demographic groups are statistically significant or due to chance, aiding fair biometric system assessment.

Contribution

The paper presents two novel statistical approaches for assessing fairness in biometric false non-match rates, addressing the gap in accounting for sampling variation.

Findings

Bootstrap hypothesis test effectively distinguishes true differences from chance.

Simulation study shows sample size and attempt correlation impact margin of error.

Methodology aids in fair decision-making in biometric systems.

Abstract

Biometric recognition is used across a variety of applications from cyber security to border security. Recent research has focused on ensuring biometric performance (false negatives and false positives) is fair across demographic groups. While there has been significant progress on the development of metrics, the evaluation of the performance across groups, and the mitigation of any problems, there has been little work incorporating statistical variation. This is important because differences among groups can be found by chance when no difference is present. In statistics this is called a Type I error. Differences among groups may be due to sampling variation or they may be due to actual difference in system performance. Discriminating between these two sources of error is essential for good decision making about fairness and equity. This paper presents two novel statistical approaches for assessing fairness across demographic groups. The first methodology is a bootstrapped-based hypothesis test, while the second is simpler test methodology focused upon non-statistical audience. For the latter we present the results of a simulation study about the relationship between the margin of error and factors such as number of subjects, number of attempts, correlation between attempts, underlying false non-match rates(FNMR's), and number of groups.