Auditing $f$-Differential Privacy in One Run

TL;DR

This paper introduces a fast, accurate auditing method for $f$-differential privacy that requires only one run of the mechanism, providing tighter privacy estimates than traditional methods.

Contribution

It presents a novel, efficient auditing procedure leveraging input randomness and $f$-DP curves to assess privacy in a single run, improving accuracy over existing approaches.

Findings

Achieves tight empirical privacy estimates.

Requires only a single run of the mechanism.

Provides more accurate privacy measurement using $f$-DP curves.

Abstract

Empirical auditing has emerged as a means of catching some of the flaws in the implementation of privacy-preserving algorithms. Existing auditing mechanisms, however, are either computationally inefficient requiring multiple runs of the machine learning algorithms or suboptimal in calculating an empirical privacy. In this work, we present a tight and efficient auditing procedure and analysis that can effectively assess the privacy of mechanisms. Our approach is efficient; similar to the recent work of Steinke, Nasr, and Jagielski (2023), our auditing procedure leverages the randomness of examples in the input dataset and requires only a single run of the target mechanism. And it is more accurate; we provide a novel analysis that enables us to achieve tight empirical privacy estimates by using the hypothesized $f$-DP curve of the mechanism, which provides a more accurate measure of privacy than the traditional $\epsilon,\delta$ differential privacy parameters. We use our auditing procure and analysis to obtain empirical privacy, demonstrating that our auditing procedure delivers tighter privacy estimates.