Fairness Issues and Mitigations in (Differentially Private) Socio-Demographic Data Processes

TL;DR

This paper examines fairness issues in socio-demographic data collection, proposing an optimization-based sampling method that accounts for privacy-preserving noise, which can unexpectedly reduce unfairness in estimates.

Contribution

It introduces a novel optimization approach for survey sampling that considers privacy effects, demonstrating how differential privacy noise can mitigate fairness biases.

Findings

Sampling errors unevenly impact group estimates, affecting fairness.

The proposed method optimizes sampling costs while controlling error margins.

Differential privacy noise can reduce unfairness by biasing smaller counts.

Abstract

Statistical agencies rely on sampling techniques to collect socio-demographic data crucial for policy-making and resource allocation. This paper shows that surveys of important societal relevance introduce sampling errors that unevenly impact group-level estimates, thereby compromising fairness in downstream decisions. To address these issues, this paper introduces an optimization approach modeled on real-world survey design processes, ensuring sampling costs are optimized while maintaining error margins within prescribed tolerances. Additionally, privacy-preserving methods used to determine sampling rates can further impact these fairness issues. This paper explores the impact of differential privacy on the statistics informing the sampling process, revealing a surprising effect: not only is the expected negative effect from the addition of noise for differential privacy negligible, but also this privacy noise can in fact reduce unfairness as it positively biases smaller counts. These findings are validated over an extensive analysis using datasets commonly applied in census statistics.