Digi2Real: Bridging the Realism Gap in Synthetic Data Face Recognition via Foundation Models

TL;DR

This paper introduces a framework that enhances the realism of synthetic face images using foundation models, significantly improving face recognition performance while addressing privacy concerns associated with real data.

Contribution

It presents a novel realism transfer method that combines graphics pipeline control with foundation model-based enhancement for synthetic face data.

Findings

Enhanced datasets lead to better face recognition accuracy.

Models trained on the improved synthetic data outperform baseline models.

The approach reduces reliance on real data for training face recognition systems.

Abstract

The accuracy of face recognition systems has improved significantly in the past few years, thanks to the large amount of data collected and advancements in neural network architectures. However, these large-scale datasets are often collected without explicit consent, raising ethical and privacy concerns. To address this, there have been proposals to use synthetic datasets for training face recognition models. Yet, such models still rely on real data to train the generative models and generally exhibit inferior performance compared to those trained on real datasets. One of these datasets, DigiFace, uses a graphics pipeline to generate different identities and intra-class variations without using real data in model training. However, the performance of this approach is poor on face recognition benchmarks, possibly due to the lack of realism in the images generated by the graphics pipeline. In this work, we introduce a novel framework for realism transfer aimed at enhancing the realism of synthetically generated face images. Our method leverages the large-scale face foundation model, and we adapt the pipeline for realism enhancement. By integrating the controllable aspects of the graphics pipeline with our realism enhancement technique, we generate a large amount of realistic variations, combining the advantages of both approaches. Our empirical evaluations demonstrate that models trained using our enhanced dataset significantly improve the performance of face recognition systems over the baseline. The source code and dataset will be publicly accessible at the following link: https://www.idiap.ch/paper/digi2real