General-Purpose $f$-DP Estimation and Auditing in a Black-Box Setting

TL;DR

This paper introduces novel black-box methods for estimating and auditing $f$-Differential Privacy ($f$-DP), providing theoretical guarantees and empirical detection of privacy violations without prior knowledge of the mechanisms.

Contribution

It presents the first black-box estimation and auditing techniques for $f$-DP, enabling validation of privacy guarantees in a general setting.

Findings

Effective estimation of $f$-DP trade-off curves.

Statistical detection of $f$-DP violations.

Validated methods on various DP mechanisms.

Abstract

In this paper we propose new methods to statistically assess $f$-Differential Privacy ($f$-DP), a recent refinement of differential privacy (DP) that remedies certain weaknesses of standard DP (including tightness under algorithmic composition). A challenge when deploying differentially private mechanisms is that DP is hard to validate, especially in the black-box setting. This has led to numerous empirical methods for auditing standard DP, while $f$-DP remains less explored. We introduce new black-box methods for $f$-DP that, unlike existing approaches for this privacy notion, do not require prior knowledge of the investigated algorithm. Our procedure yields a complete estimate of the $f$-DP trade-off curve, with theoretical guarantees of convergence. Additionally, we propose an efficient auditing method that empirically detects $f$-DP violations with statistical certainty, merging techniques from non-parametric estimation and optimal classification theory. Through experiments on a range of DP mechanisms, we demonstrate the effectiveness of our estimation and auditing procedures.