AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmark

TL;DR

This paper introduces AI-Face, a large-scale dataset of demographically annotated AI-generated faces, and provides a fairness benchmark to evaluate and improve AI face detectors across diverse demographic groups.

Contribution

The paper presents the first million-scale demographically annotated AI-generated face dataset and a comprehensive fairness benchmark for AI face detection methods.

Findings

Current detectors show biased performance across demographics

The dataset enables development of fairer AI face detection algorithms

Benchmark results reveal key challenges in demographic fairness

Abstract

AI-generated faces have enriched human life, such as entertainment, education, and art. However, they also pose misuse risks. Therefore, detecting AI-generated faces becomes crucial, yet current detectors show biased performance across different demographic groups. Mitigating biases can be done by designing algorithmic fairness methods, which usually require demographically annotated face datasets for model training. However, no existing dataset encompasses both demographic attributes and diverse generative methods simultaneously, which hinders the development of fair detectors for AI-generated faces. In this work, we introduce the AI-Face dataset, the first million-scale demographically annotated AI-generated face image dataset, including real faces, faces from deepfake videos, and faces generated by Generative Adversarial Networks and Diffusion Models. Based on this dataset, we conduct the first comprehensive fairness benchmark to assess various AI face detectors and provide valuable insights and findings to promote the future fair design of AI face detectors. Our AI-Face dataset and benchmark code are publicly available at https://github.com/Purdue-M2/AI-Face-FairnessBench