The Hitchhiker's Guide to Efficient, End-to-End, and Tight DP Auditing

TL;DR

This paper reviews and systematizes differential privacy auditing techniques, focusing on efficiency, end-to-end process, and tightness, and identifies open challenges and future research directions.

Contribution

It introduces a comprehensive framework for evaluating DP audits and analyzes current methods to highlight limitations and open problems.

Findings

Highlights overlooked details in current DP auditing techniques.

Identifies key limiting factors to achieving efficiency, end-to-end-ness, and tightness.

Provides a systematic methodology for assessing progress in DP auditing.

Abstract

In this paper, we systematize research on auditing Differential Privacy (DP) techniques, aiming to identify key insights and open challenges. First, we introduce a comprehensive framework for reviewing work in the field and establish three cross-contextual desiderata that DP audits should target -- namely, efficiency, end-to-end-ness, and tightness. Then, we systematize the modes of operation of state-of-the-art DP auditing techniques, including threat models, attacks, and evaluation functions. This allows us to highlight key details overlooked by prior work, analyze the limiting factors to achieving the three desiderata, and identify open research problems. Overall, our work provides a reusable and systematic methodology geared to assess progress in the field and identify friction points and future directions for our community to focus on.