Estimating and Implementing Conventional Fairness Metrics With Probabilistic Protected Features

TL;DR

This paper introduces methods to estimate and mitigate fairness violations in models when protected attribute labels are scarce, using probabilistic estimates and contextual information to achieve tighter fairness bounds and improved fairness-accuracy trade-offs.

Contribution

It develops novel techniques to measure and reduce fairness violations with limited protected attribute data by leveraging probabilistic estimates and contextual relationships.

Findings

Measurement bounds are up to 5.5x tighter than previous methods.

The proposed training method reduces disparity effectively.

Lesser fairness-accuracy trade-offs compared to existing approaches.

Abstract

The vast majority of techniques to train fair models require access to the protected attribute (e.g., race, gender), either at train time or in production. However, in many important applications this protected attribute is largely unavailable. In this paper, we develop methods for measuring and reducing fairness violations in a setting with limited access to protected attribute labels. Specifically, we assume access to protected attribute labels on a small subset of the dataset of interest, but only probabilistic estimates of protected attribute labels (e.g., via Bayesian Improved Surname Geocoding) for the rest of the dataset. With this setting in mind, we propose a method to estimate bounds on common fairness metrics for an existing model, as well as a method for training a model to limit fairness violations by solving a constrained non-convex optimization problem. Unlike similar existing approaches, our methods take advantage of contextual information -- specifically, the relationships between a model's predictions and the probabilistic prediction of protected attributes, given the true protected attribute, and vice versa -- to provide tighter bounds on the true disparity. We provide an empirical illustration of our methods using voting data. First, we show our measurement method can bound the true disparity up to 5.5x tighter than previous methods in these applications. Then, we demonstrate that our training technique effectively reduces disparity while incurring lesser fairness-accuracy trade-offs than other fair optimization methods with limited access to protected attributes.