Private Algorithms for Stochastic Saddle Points and Variational Inequalities: Beyond Euclidean Geometry

TL;DR

This paper develops differentially private algorithms for stochastic saddle point problems and variational inequalities in both Euclidean and non-Euclidean geometries, achieving near-optimal convergence rates with broad applicability.

Contribution

It introduces a general DP algorithm for SSPs and SVIs in non-Euclidean setups, with new analysis tools and near-optimal convergence guarantees.

Findings

Achieves near-optimal rate of rac{rac{1}{\u221a{n}} + rac{rac{rac{rac{d}{npsilon}}} for SSPs in rac{rac{1}{rac{n}} + rac{rac{rac{d}{npsilon}}} for SVIs.

Provides a general DP algorithm for rac{rac{p,q}{p,qor} rac{rac{1,2}{1,2} ases, overing non-Euclidean geometries.

Develops new analytical tools for recursive regularization and generalization in the context of DP SSPs and SVIs.

Abstract

In this work, we conduct a systematic study of stochastic saddle point problems (SSP) and stochastic variational inequalities (SVI) under the constraint of $(\epsilon,\delta)$-differential privacy (DP) in both Euclidean and non-Euclidean setups. We first consider Lipschitz convex-concave SSPs in the $\ell_p/\ell_q$ setup, $p,q\in[1,2]$. Here, we obtain a bound of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$ on the strong SP-gap, where $n$ is the number of samples and $d$ is the dimension. This rate is nearly optimal for any $p,q\in[1,2]$. Without additional assumptions, such as smoothness or linearity requirements, prior work under DP has only obtained this rate when $p=q=2$ (i.e., only in the Euclidean setup). Further, existing algorithms have each only been shown to work for specific settings of $p$ and $q$ and under certain assumptions on the loss and the feasible set, whereas we provide a general algorithm for DP SSPs whenever $p,q\in[1,2]$. Our result is obtained via a novel analysis of the recursive regularization algorithm. In particular, we develop new tools for analyzing generalization, which may be of independent interest. Next, we turn our attention towards SVIs with a monotone, bounded and Lipschitz operator and consider $\ell_p$-setups, $p\in[1,2]$. Here, we provide the first analysis which obtains a bound on the strong VI-gap of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$. For $p-1=\Omega(1)$, this rate is near optimal due to existing lower bounds. To obtain this result, we develop a modified version of recursive regularization. Our analysis builds on the techniques we develop for SSPs as well as employing additional novel components which handle difficulties arising from adapting the recursive regularization framework to SVIs.