Synthetic Counterfactual Faces

TL;DR

This paper introduces a generative AI framework to create high-quality synthetic counterfactual face data, aiding in evaluating and understanding the robustness and fairness of face recognition systems against distributional shifts.

Contribution

The paper presents a novel AI-based pipeline for generating targeted synthetic face data to improve robustness testing and explainability of computer vision models.

Findings

Effective face generation pipeline validated with user studies

Identified facial attributes causing model failures

Demonstrated pipeline's utility on commercial vision models

Abstract

Computer vision systems have been deployed in various applications involving biometrics like human faces. These systems can identify social media users, search for missing persons, and verify identity of individuals. While computer vision models are often evaluated for accuracy on available benchmarks, more annotated data is necessary to learn about their robustness and fairness against semantic distributional shifts in input data, especially in face data. Among annotated data, counterfactual examples grant strong explainability characteristics. Because collecting natural face data is prohibitively expensive, we put forth a generative AI-based framework to construct targeted, counterfactual, high-quality synthetic face data. Our synthetic data pipeline has many use cases, including face recognition systems sensitivity evaluations and image understanding system probes. The pipeline is validated with multiple user studies. We showcase the efficacy of our face generation pipeline on a leading commercial vision model. We identify facial attributes that cause vision systems to fail.