Estimating Structural Disparities for Face Models

TL;DR

This paper investigates methods to estimate disparities in face model performance across groups without explicit group labels, using face embeddings as proxies, and evaluates this approach on face attribute and affect prediction tasks.

Contribution

It introduces a novel approach to measure model disparities using proxy embeddings when group labels are unavailable or sensitive.

Findings

Embeddings from face recognition models can serve as effective proxies for group disparity analysis.

The proposed method enables disparity measurement in scenarios lacking explicit group labels.

Experiments demonstrate meaningful disparity estimates using proxy embeddings in face-related tasks.

Abstract

In machine learning, disparity metrics are often defined by measuring the difference in the performance or outcome of a model, across different sub-populations (groups) of datapoints. Thus, the inputs to disparity quantification consist of a model's predictions $\hat{y}$, the ground-truth labels for the predictions $y$, and group labels $g$ for the data points. Performance of the model for each group is calculated by comparing $\hat{y}$ and $y$ for the datapoints within a specific group, and as a result, disparity of performance across the different groups can be calculated. In many real world scenarios however, group labels ($g$) may not be available at scale during training and validation time, or collecting them might not be feasible or desirable as they could often be sensitive information. As a result, evaluating disparity metrics across categorical groups would not be feasible. On the other hand, in many scenarios noisy groupings may be obtainable using some form of a proxy, which would allow measuring disparity metrics across sub-populations. Here we explore performing such analysis on computer vision models trained on human faces, and on tasks such as face attribute prediction and affect estimation. Our experiments indicate that embeddings resulting from an off-the-shelf face recognition model, could meaningfully serve as a proxy for such estimation.