Efficient Public Verification of Private ML via Regularization

TL;DR

This paper introduces a new differentially private stochastic convex optimization algorithm that allows for efficient verification of privacy guarantees, reducing computational costs compared to training, especially on large datasets.

Contribution

It presents the first DP-SCO algorithm with near optimal privacy-utility trade-offs that can be verified more efficiently than the training process.

Findings

Verification cost is lower than training cost for large datasets.

Achieves tight privacy-utility trade-offs using regularized objectives.

Verification scales better than training, reducing overall computational effort.

Abstract

Training with differential privacy (DP) provides a guarantee to members in a dataset that they cannot be identified by users of the released model. However, those data providers, and, in general, the public, lack methods to efficiently verify that models trained on their data satisfy DP guarantees. The amount of compute needed to verify DP guarantees for current algorithms scales with the amount of compute required to train the model. In this paper we design the first DP algorithm with near optimal privacy-utility trade-offs but whose DP guarantees can be verified cheaper than training. We focus on DP stochastic convex optimization (DP-SCO), where optimal privacy-utility trade-offs are known. Here we show we can obtain tight privacy-utility trade-offs by privately minimizing a series of regularized objectives and only using the standard DP composition bound. Crucially, this method can be verified with much less compute than training. This leads to the first known DP-SCO algorithm with near optimal privacy-utility whose DP verification scales better than training cost, significantly reducing verification costs on large datasets.