Enhancing One-run Privacy Auditing with Quantile Regression-Based Membership Inference

TL;DR

This paper improves privacy auditing for differentially private models in black-box settings by using quantile regression-based membership inference, achieving tighter bounds efficiently.

Contribution

It introduces a novel quantile regression-based membership inference attack to enhance one-run privacy auditing in black-box scenarios.

Findings

Achieves tighter privacy bounds on DP-SGD models.

Maintains computational efficiency of one-run auditing.

Effective on CIFAR-10 image classification models.

Abstract

Differential privacy (DP) auditing aims to provide empirical lower bounds on the privacy guarantees of DP mechanisms like DP-SGD. While some existing techniques require many training runs that are prohibitively costly, recent work introduces one-run auditing approaches that effectively audit DP-SGD in white-box settings while still being computationally efficient. However, in the more practical black-box setting where gradients cannot be manipulated during training and only the last model iterate is observed, prior work shows that there is still a large gap between the empirical lower bounds and theoretical upper bounds. Consequently, in this work, we study how incorporating approaches for stronger membership inference attacks (MIA) can improve one-run auditing in the black-box setting. Evaluating on image classification models trained on CIFAR-10 with DP-SGD, we demonstrate that our proposed approach, which utilizes quantile regression for MIA, achieves tighter bounds while crucially maintaining the computational efficiency of one-run methods.