Federated Measurement of Demographic Disparities from Quantile Sketches

TL;DR

This paper introduces a federated approach to measure demographic disparities using score distributions, enabling privacy-preserving auditing of fairness metrics across siloed data sources with theoretical guarantees and practical efficiency.

Contribution

It develops a federated protocol for estimating population-level demographic parity via score distributions, with a novel Wasserstein--Frechet variance measure and theoretical bounds.

Findings

Few dozen quantiles recover global disparity accurately

Protocol achieves $O(1/k)$ discretization bias

Experiments validate effectiveness on synthetic and real data

Abstract

Many fairness goals are defined at a population level that misaligns with siloed data collection, which remains unsharable due to privacy regulations. Horizontal federated learning (FL) enables collaborative modeling across clients with aligned features without sharing raw data. We study federated auditing of demographic parity through score distributions, measuring disparity as a Wasserstein--Frechet variance between sensitive-group score laws, and expressing the population metric in federated form that makes explicit how silo-specific selection drives local-global mismatch. For the squared Wasserstein distance, we prove an ANOVA-style decomposition that separates (i) selection-induced mixture effects from (ii) cross-silo heterogeneity, yielding tight bounds linking local and global metrics. We then propose a one-shot, communication-efficient protocol in which each silo shares only group counts and a quantile summary of its local score distributions, enabling the server to estimate global disparity and its decomposition, with $O(1/k)$ discretization bias ($k$ quantiles) and finite-sample guarantees. Experiments on synthetic data and COMPAS show that a few dozen quantiles suffice to recover global disparity and diagnose its sources.