On the "Illusion" of Gender Bias in Face Recognition: Explaining the Fairness Issue Through Non-demographic Attributes

TL;DR

This paper investigates gender bias in face recognition systems by analyzing non-demographic facial attributes, revealing that bias can be mitigated through specific attribute combinations, challenging biological assumptions and informing fairness improvements.

Contribution

It introduces a decorrelation and aggregation toolchain, new fairness metrics, and an unsupervised algorithm to identify attribute sets that eliminate gender bias in face recognition.

Findings

Gender bias diminishes with specific attribute combinations.

Bias is linked to social rather than biological factors.

Method enables balanced testing datasets for fairness analysis.

Abstract

Face recognition systems (FRS) exhibit significant accuracy differences based on the user's gender. Since such a gender gap reduces the trustworthiness of FRS, more recent efforts have tried to find the causes. However, these studies make use of manually selected, correlated, and small-sized sets of facial features to support their claims. In this work, we analyse gender bias in face recognition by successfully extending the search domain to decorrelated combinations of 40 non-demographic facial characteristics. First, we propose a toolchain to effectively decorrelate and aggregate facial attributes to enable a less-biased gender analysis on large-scale data. Second, we introduce two new fairness metrics to measure fairness with and without context. Based on these grounds, we thirdly present a novel unsupervised algorithm able to reliably identify attribute combinations that lead to vanishing bias when used as filter predicates for balanced testing datasets. The experiments show that the gender gap vanishes when images of male and female subjects share specific attributes, clearly indicating that the issue is not a question of biology but of the social definition of appearance. These findings could reshape our understanding of fairness in face biometrics and provide insights into FRS, helping to address gender bias issues.