Optimal Guarantees for Auditing R\'enyi Differentially Private Machine Learning

TL;DR

This paper introduces an optimal, statistically rigorous framework for auditing Rènyi differential privacy in machine learning, providing theoretical guarantees and empirical improvements over prior methods.

Contribution

It develops a hypothesis testing-based auditing framework with explicit confidence intervals and proves its information-theoretic optimality for RDP guarantees.

Findings

Achieves tight confidence intervals for RDP estimation.

Establishes minimax lower bounds matching the sample complexity.

Demonstrates empirical improvements on MNIST and CIFAR-10 datasets.

Abstract

We study black-box auditing for machine learning algorithms that claim R \ 'enyi differential privacy (RDP) guarantees. We introduce an auditing framework, based on hypothesis testing, that directly estimates R\'enyi divergence between neighboring executions using the Donsker-Varadhan (DV) variational estimator. Our analysis yields explicit and non-asymptotic confidence intervals for RDP auditing via class-restricted DV estimators, separating statistical estimation error from algorithmic privacy leakage. We prove matching minimax lower bounds showing that, up to logarithmic factors, our sample-complexity guarantees are information-theoretically optimal, thereby establishing the first optimal guarantees for auditing RDP via DV estimators. Empirically, we instantiate our framework for auditing DP-SGD in a fully black-box setting. Across MNIST and CIFAR-10, and over a wide range of privacy regimes, our auditors produce a strong overall improvement on empirical RDP lower bounds compared to prior state-of-the-art black-box methods especially at small and moderate R\'enyi orders where accurate auditing is most challenging.