Revealing the True Cost of Locally Differentially Private Protocols: An Auditing Perspective

TL;DR

This paper introduces LDP-Auditor, a framework for empirically auditing local differential privacy protocols, revealing privacy loss factors, and identifying bugs in existing implementations, thus aiding practitioners in better privacy parameter selection.

Contribution

The paper presents the first empirical auditing framework for Local Differential Privacy, analyzing privacy loss factors and uncovering issues in current LDP tools.

Findings

Identified key factors affecting local privacy loss.

Discovered a bug in a state-of-the-art LDP Python package.

Provided insights into privacy attack vectors and parameter impacts.

Abstract

While the existing literature on Differential Privacy (DP) auditing predominantly focuses on the centralized model (e.g., in auditing the DP-SGD algorithm), we advocate for extending this approach to audit Local DP (LDP). To achieve this, we introduce the LDP-Auditor framework for empirically estimating the privacy loss of locally differentially private mechanisms. This approach leverages recent advances in designing privacy attacks against LDP frequency estimation protocols. More precisely, through the analysis of numerous state-of-the-art LDP protocols, we extensively explore the factors influencing the privacy audit, such as the impact of different encoding and perturbation functions. Additionally, we investigate the influence of the domain size and the theoretical privacy loss parameters $\epsilon$ and $\delta$ on local privacy estimation. In-depth case studies are also conducted to explore specific aspects of LDP auditing, including distinguishability attacks on LDP protocols for longitudinal studies and multidimensional data. Finally, we present a notable achievement of our LDP-Auditor framework, which is the discovery of a bug in a state-of-the-art LDP Python package. Overall, our LDP-Auditor framework as well as our study offer valuable insights into the sources of randomness and information loss in LDP protocols. These contributions collectively provide a realistic understanding of the local privacy loss, which can help practitioners in selecting the LDP mechanism and privacy parameters that best align with their specific requirements. We open-sourced LDP-Auditor in \url{https://github.com/hharcolezi/ldp-audit}.