Detecting Violations of Differential Privacy

TL;DR

This paper presents a statistical method for detecting violations of differential privacy in algorithms by generating short, understandable counterexamples, aiding developers in debugging privacy-preserving algorithms effectively.

Contribution

It introduces a novel statistical approach to identify privacy violations and generate human-understandable counterexamples for incorrect differential privacy algorithms.

Findings

Successfully rejects incorrect algorithms within seconds

Generates understandable counterexamples for privacy violations

Validates approach on various published algorithms

Abstract

The widespread acceptance of differential privacy has led to the publication of many sophisticated algorithms for protecting privacy. However, due to the subtle nature of this privacy definition, many such algorithms have bugs that make them violate their claimed privacy. In this paper, we consider the problem of producing counterexamples for such incorrect algorithms. The counterexamples are designed to be short and human-understandable so that the counterexample generator can be used in the development process -- a developer could quickly explore variations of an algorithm and investigate where they break down. Our approach is statistical in nature. It runs a candidate algorithm many times and uses statistical tests to try to detect violations of differential privacy. An evaluation on a variety of incorrect published algorithms validates the usefulness of our approach: it correctly rejects incorrect algorithms and provides counterexamples for them within a few seconds.