Numerical Composition of Differential Privacy

TL;DR

This paper introduces a fast, efficient algorithm for composing differential privacy guarantees, significantly reducing computation time while maintaining accuracy, which enhances privacy analysis for DP algorithms like DP-SGD.

Contribution

The paper presents a novel algorithm based on privacy loss random variables that improves the computational complexity of privacy composition from $\tilde{\Omega}(k^{1.5})$ to $\tilde{O}(\sqrt{k})$, enabling faster privacy calculations.

Findings

The new algorithm accurately computes the privacy loss of DP-SGD.

It speeds up privacy computations by several orders of magnitude.

The method maintains accuracy comparable to previous approaches.

Abstract

We give a fast algorithm to optimally compose privacy guarantees of differentially private (DP) algorithms to arbitrary accuracy. Our method is based on the notion of privacy loss random variables to quantify the privacy loss of DP algorithms. The running time and memory needed for our algorithm to approximate the privacy curve of a DP algorithm composed with itself $k$ times is $\tilde{O}(\sqrt{k})$. This improves over the best prior method by Koskela et al. (2020) which requires $\tilde{\Omega}(k^{1.5})$ running time. We demonstrate the utility of our algorithm by accurately computing the privacy loss of DP-SGD algorithm of Abadi et al. (2016) and showing that our algorithm speeds up the privacy computations by a few orders of magnitude compared to prior work, while maintaining similar accuracy.