What Should Be Balanced in a "Balanced" Face Recognition Dataset?

TL;DR

This paper examines the limitations of current 'balanced' face recognition datasets and introduces a bias-aware toolkit to create more comprehensive evaluation datasets considering multiple factors affecting accuracy.

Contribution

It highlights the inadequacy of balancing only identities and images and proposes a toolkit for creating more balanced datasets across various influential factors.

Findings

Balancing identities and images alone does not ensure fairness.

Current datasets often overlook factors like pose, brightness, and quality.

A new toolkit helps create more balanced and less biased evaluation datasets.

Abstract

The issue of demographic disparities in face recognition accuracy has attracted increasing attention in recent years. Various face image datasets have been proposed as 'fair' or 'balanced' to assess the accuracy of face recognition algorithms across demographics. These datasets typically balance the number of identities and images across demographics. It is important to note that the number of identities and images in an evaluation dataset are {\em not} driving factors for 1-to-1 face matching accuracy. Moreover, balancing the number of identities and images does not ensure balance in other factors known to impact accuracy, such as head pose, brightness, and image quality. We demonstrate these issues using several recently proposed datasets. To improve the ability to perform less biased evaluations, we propose a bias-aware toolkit that facilitates creation of cross-demographic evaluation datasets balanced on factors mentioned in this paper.