DP-Auditorium: a Large Scale Library for Auditing Differential Privacy

TL;DR

DP-Auditorium is a comprehensive, flexible library that advances the testing of differential privacy mechanisms by introducing new algorithms and evaluation strategies, addressing the challenge of verifying privacy guarantees effectively.

Contribution

The paper presents DP-Auditorium, a modular library with novel algorithms for distribution distance estimation, enhancing the testing and verification of differential privacy mechanisms.

Findings

Multiple testing techniques are necessary for reliable privacy verification.

The library's algorithms provide new estimation guarantees tailored for privacy testing.

No single tester outperforms others across all scenarios.

Abstract

New regulations and increased awareness of data privacy have led to the deployment of new and more efficient differentially private mechanisms across public institutions and industries. Ensuring the correctness of these mechanisms is therefore crucial to ensure the proper protection of data. However, since differential privacy is a property of the mechanism itself, and not of an individual output, testing whether a mechanism is differentially private is not a trivial task. While ad hoc testing techniques exist under specific assumptions, no concerted effort has been made by the research community to develop a flexible and extendable tool for testing differentially private mechanisms. This paper introduces DP-Auditorium as a step advancing research in this direction. DP-Auditorium abstracts the problem of testing differential privacy into two steps: (1) measuring the distance between distributions, and (2) finding neighboring datasets where a mechanism generates output distributions maximizing such distance. From a technical point of view, we propose three new algorithms for evaluating the distance between distributions. While these algorithms are well-established in the statistics community, we provide new estimation guarantees that exploit the fact that we are only interested in verifying whether a mechanism is differentially private, and not in obtaining an exact estimate of the distance between two distributions. DP-Auditorium is easily extensible, as demonstrated in this paper by implementing a well-known approximate differential privacy testing algorithm into our library. We provide an extensive comparison to date of multiple testers across varying sample sizes and differential privacy parameters, demonstrating that there is no single tester that dominates all others, and that a combination of different techniques is required to ensure proper testing of mechanisms.