Reliable and Reproducible Demographic Inference for Fairness in Face Analysis

TL;DR

This paper introduces a reproducible demographic inference pipeline for face analysis fairness auditing, demonstrating improved reliability and robustness, especially for ethnicity, through a modular transfer learning approach and comprehensive benchmarking.

Contribution

It proposes a modular transfer learning-based demographic inference pipeline that enhances reliability and robustness for fairness auditing in face analysis systems.

Findings

Outperforms strong baselines on ethnicity inference

Improves fairness estimates with more reliable DAI

Introduces a robustness metric for demographic inference

Abstract

Fairness evaluation in face analysis systems (FAS) typically depends on automatic demographic attribute inference (DAI), which itself relies on predefined demographic segmentation. However, the validity of fairness auditing hinges on the reliability of the DAI process. We begin by providing a theoretical motivation for this dependency, showing that improved DAI reliability leads to less biased and lower-variance estimates of FAS fairness. To address this, we propose a fully reproducible DAI pipeline that replaces conventional end-to-end training with a modular transfer learning approach. Our design integrates pretrained face recognition encoders with non-linear classification heads. We audit this pipeline across three dimensions: accuracy, fairness, and a newly introduced notion of robustness, defined via intra-identity consistency. The proposed robustness metric is applicable to any demographic segmentation scheme. We benchmark the pipeline on gender and ethnicity inference across multiple datasets and training setups. Our results show that the proposed method outperforms strong baselines, particularly on ethnicity, which is the more challenging attribute. To promote transparency and reproducibility, we will publicly release the training dataset metadata, full codebase, pretrained models, and evaluation toolkit. This work contributes a reliable foundation for demographic inference in fairness auditing.