Limitations of Face Image Generation

TL;DR

This paper critically examines the limitations of face image generation by state-of-the-art diffusion models, highlighting issues like demographic bias, prompt faithfulness, and data-driven distributional shifts, using a comprehensive evaluation framework.

Contribution

It introduces a novel framework for auditing generated face images, combining qualitative, quantitative, and user studies, and provides an analytical model linking training data to model performance.

Findings

Identified demographic disparities in generated faces

Highlighted challenges in faithfulness to text prompts

Analyzed how training data influences generation quality

Abstract

Text-to-image diffusion models have achieved widespread popularity due to their unprecedented image generation capability. In particular, their ability to synthesize and modify human faces has spurred research into using generated face images in both training data augmentation and model performance assessments. In this paper, we study the efficacy and shortcomings of generative models in the context of face generation. Utilizing a combination of qualitative and quantitative measures, including embedding-based metrics and user studies, we present a framework to audit the characteristics of generated faces conditioned on a set of social attributes. We applied our framework on faces generated through state-of-the-art text-to-image diffusion models. We identify several limitations of face image generation that include faithfulness to the text prompt, demographic disparities, and distributional shifts. Furthermore, we present an analytical model that provides insights into how training data selection contributes to the performance of generative models.