Towards Fair Face Verification: An In-depth Analysis of Demographic Biases

TL;DR

This paper analyzes demographic biases in face verification systems, revealing significant disparities across race, age, and gender, especially at intersectional levels, and emphasizes the need for comprehensive fairness metrics beyond accuracy.

Contribution

It provides an in-depth intersectional bias analysis in face verification, incorporating multiple fairness metrics, and highlights pervasive demographic disparities in current systems.

Findings

Africans have 11.25% lower TPR than Caucasians.

Intersectional groups like African females over 60 show 39.89% higher mistreatment.

Biases extend beyond race, affecting age and gender groups.

Abstract

Deep learning-based person identification and verification systems have remarkably improved in terms of accuracy in recent years; however, such systems, including widely popular cloud-based solutions, have been found to exhibit significant biases related to race, age, and gender, a problem that requires in-depth exploration and solutions. This paper presents an in-depth analysis, with a particular emphasis on the intersectionality of these demographic factors. Intersectional bias refers to the performance discrepancies w.r.t. the different combinations of race, age, and gender groups, an area relatively unexplored in current literature. Furthermore, the reliance of most state-of-the-art approaches on accuracy as the principal evaluation metric often masks significant demographic disparities in performance. To counter this crucial limitation, we incorporate five additional metrics in our quantitative analysis, including disparate impact and mistreatment metrics, which are typically ignored by the relevant fairness-aware approaches. Results on the Racial Faces in-the-Wild (RFW) benchmark indicate pervasive biases in face recognition systems, extending beyond race, with different demographic factors yielding significantly disparate outcomes. In particular, Africans demonstrate an 11.25% lower True Positive Rate (TPR) compared to Caucasians, while only a 3.51% accuracy drop is observed. Even more concerning, the intersections of multiple protected groups, such as African females over 60 years old, demonstrate a +39.89% disparate mistreatment rate compared to the highest Caucasians rate. By shedding light on these biases and their implications, this paper aims to stimulate further research towards developing fairer, more equitable face recognition and verification systems.