Auditing Private Prediction

TL;DR

This paper introduces a novel framework for empirically auditing the privacy leakage of private prediction algorithms, revealing that some algorithms are more vulnerable to poisoning and query control attacks, and suggesting improvements in privacy analysis.

Contribution

It presents the first framework for auditing private prediction algorithms using adversaries with varying capabilities and introduces techniques to evaluate privacy leakage in terms of Renyi DP.

Findings

Privacy analysis of private prediction can be improved.

Easier-to-poison algorithms exhibit higher privacy leakage.

Privacy leakage is lower for adversaries without query control.

Abstract

Differential privacy (DP) offers a theoretical upper bound on the potential privacy leakage of analgorithm, while empirical auditing establishes a practical lower bound. Auditing techniques exist forDP training algorithms. However machine learning can also be made private at inference. We propose thefirst framework for auditing private prediction where we instantiate adversaries with varying poisoningand query capabilities. This enables us to study the privacy leakage of four private prediction algorithms:PATE [Papernot et al., 2016], CaPC [Choquette-Choo et al., 2020], PromptPATE [Duan et al., 2023],and Private-kNN [Zhu et al., 2020]. To conduct our audit, we introduce novel techniques to empiricallyevaluate privacy leakage in terms of Renyi DP. Our experiments show that (i) the privacy analysis ofprivate prediction can be improved, (ii) algorithms which are easier to poison lead to much higher privacyleakage, and (iii) the privacy leakage is significantly lower for adversaries without query control than thosewith full control.