Curator Attack: When Blackbox Differential Privacy Auditing Loses Its Power

TL;DR

This paper reveals fundamental flaws in blackbox differential privacy auditing methods, showing they can be deceived by ignoring small probabilities, which risks overestimating privacy guarantees in data sanitization.

Contribution

It identifies a critical false positive issue in blackbox auditing for differential privacy, backed by theoretical analysis and experimental validation, highlighting the need for more reliable methods.

Findings

Blackbox auditors can be fooled by small probability events.

Current blackbox auditing tools may overstate privacy guarantees.

Experimental validation demonstrates practical threats to differential privacy mechanisms.

Abstract

A surge in data-driven applications enhances everyday life but also raises serious concerns about private information leakage. Hence many privacy auditing tools are emerging for checking if the data sanitization performed meets the privacy standard of the data owner. Blackbox auditing for differential privacy is particularly gaining popularity for its effectiveness and applicability to a wide range of scenarios. Yet, we identified that blackbox auditing is essentially flawed with its setting: small probabilities or densities are ignored due to inaccurate observation. Our argument is based on a solid false positive analysis from a hypothesis testing perspective, which is missed out by prior blackbox auditing tools. This oversight greatly reduces the reliability of these tools, as it allows malicious or incapable data curators to pass the auditing with an overstated privacy guarantee, posing significant risks to data owners. We demonstrate the practical existence of such threats in classical differential privacy mechanisms against four representative blackbox auditors with experimental validations. Our findings aim to reveal the limitations of blackbox auditing tools, empower the data owner with the awareness of risks in using these tools, and encourage the development of more reliable differential privacy auditing methods.