On Measuring Fairness in Generative Models

TL;DR

This paper critically examines fairness measurement in generative models, revealing significant measurement errors in existing methods, and introduces CLEAM, a new framework that improves accuracy and uncovers biases in popular models.

Contribution

The paper identifies flaws in current fairness measurement methods and proposes CLEAM, a novel error-aware framework that enhances measurement accuracy in generative models.

Findings

Existing fairness metrics have high measurement errors.

CLEAM significantly reduces measurement errors.

Biases are prevalent in popular text-to-image and GAN models.

Abstract

Recently, there has been increased interest in fair generative models. In this work, we conduct, for the first time, an in-depth study on fairness measurement, a critical component in gauging progress on fair generative models. We make three contributions. First, we conduct a study that reveals that the existing fairness measurement framework has considerable measurement errors, even when highly accurate sensitive attribute (SA) classifiers are used. These findings cast doubts on previously reported fairness improvements. Second, to address this issue, we propose CLassifier Error-Aware Measurement (CLEAM), a new framework which uses a statistical model to account for inaccuracies in SA classifiers. Our proposed CLEAM reduces measurement errors significantly, e.g., 4.98% $\rightarrow$ 0.62% for StyleGAN2 w.r.t. Gender. Additionally, CLEAM achieves this with minimal additional overhead. Third, we utilize CLEAM to measure fairness in important text-to-image generator and GANs, revealing considerable biases in these models that raise concerns about their applications. Code and more resources: https://sutd-visual-computing-group.github.io/CLEAM/.