The Impact of Balancing Real and Synthetic Data on Accuracy and Fairness in Face Recognition

TL;DR

This study examines how combining demographically balanced authentic and synthetic data affects face recognition accuracy and fairness, highlighting diffusion-based synthetic data's benefits for accuracy but limited impact on fairness.

Contribution

It demonstrates the effectiveness of diffusion-based synthetic data in improving face recognition accuracy and evaluates their limited influence on fairness when combined with authentic data.

Findings

Diffusion-based synthetic data improves recognition accuracy.

Balanced data has minimal impact on fairness.

Combining synthetic and authentic data can enhance model performance.

Abstract

Over the recent years, the advancements in deep face recognition have fueled an increasing demand for large and diverse datasets. Nevertheless, the authentic data acquired to create those datasets is typically sourced from the web, which, in many cases, can lead to significant privacy issues due to the lack of explicit user consent. Furthermore, obtaining a demographically balanced, large dataset is even more difficult because of the natural imbalance in the distribution of images from different demographic groups. In this paper, we investigate the impact of demographically balanced authentic and synthetic data, both individually and in combination, on the accuracy and fairness of face recognition models. Initially, several generative methods were used to balance the demographic representations of the corresponding synthetic datasets. Then a state-of-the-art face encoder was trained and evaluated using (combinations of) synthetic and authentic images. Our findings emphasized two main points: (i) the increased effectiveness of training data generated by diffusion-based models in enhancing accuracy, whether used alone or combined with subsets of authentic data, and (ii) the minimal impact of incorporating balanced data from pre-trained generative methods on fairness (in nearly all tested scenarios using combined datasets, fairness scores remained either unchanged or worsened, even when compared to unbalanced authentic datasets). Source code and data are available at \url{https://cutt.ly/AeQy1K5G} for reproducibility.