Fairness on Synthetic Visual and Thermal Mask Images

TL;DR

This paper evaluates fairness in biometric systems using synthetic visual and thermal mask images, highlighting dataset bias impacts and proposing assessment methods applicable to both real and synthetic data.

Contribution

It introduces a process to assess fairness on synthetic images and demonstrates how synthetic data can mitigate class imbalance in biometric datasets.

Findings

Demographic parity difference of 1.59 at random guessing

Fairness decreases as recognition performance increases

Synthetic data augmentation reduces bias in datasets

Abstract

In this paper, we study performance and fairness on visual and thermal images and expand the assessment to masked synthetic images. Using the SpeakingFace and Thermal-Mask dataset, we propose a process to assess fairness on real images and show how the same process can be applied to synthetic images. The resulting process shows a demographic parity difference of 1.59 for random guessing and increases to 5.0 when the recognition performance increases to a precision and recall rate of 99.99\%. We indicate that inherently biased datasets can deeply impact the fairness of any biometric system. A primary cause of a biased dataset is the class imbalance due to the data collection process. To address imbalanced datasets, the classes with fewer samples can be augmented with synthetic images to generate a more balanced dataset resulting in less bias when training a machine learning system. For biometric-enabled systems, fairness is of critical importance, while the related concept of Equity, Diversity, and Inclusion (EDI) is well suited for the generalization of fairness in biometrics, in this paper, we focus on the 3 most common demographic groups age, gender, and ethnicity.