Private Stochastic Convex Optimization with Optimal Rates

TL;DR

This paper establishes that differentially private algorithms for stochastic convex optimization can achieve optimal excess population loss rates comparable to non-private algorithms, resolving a significant gap in understanding privacy-utility trade-offs.

Contribution

It proves that the optimal excess population loss for DP algorithms matches the larger of the non-private excess loss and the empirical excess loss, with rates of 1/√n, improving upon previous 1/n^{1/4} results.

Findings

DP algorithms can match non-private excess loss rates of 1/√n.

The analysis uses algorithmic stability to connect privacy, generalization, and utility.

Optimal rates are achieved up to logarithmic factors.

Abstract

We study differentially private (DP) algorithms for stochastic convex optimization (SCO). In this problem the goal is to approximately minimize the population loss given i.i.d. samples from a distribution over convex and Lipschitz loss functions. A long line of existing work on private convex optimization focuses on the empirical loss and derives asymptotically tight bounds on the excess empirical loss. However a significant gap exists in the known bounds for the population loss. We show that, up to logarithmic factors, the optimal excess population loss for DP algorithms is equal to the larger of the optimal non-private excess population loss, and the optimal excess empirical loss of DP algorithms. This implies that, contrary to intuition based on private ERM, private SCO has asymptotically the same rate of $1/\sqrt{n}$ as non-private SCO in the parameter regime most common in practice. The best previous result in this setting gives rate of $1/n^{1/4}$. Our approach builds on existing differentially private algorithms and relies on the analysis of algorithmic stability to ensure generalization.