# Differentially Private Algorithms for the Stochastic Saddle Point Problem with Optimal Rates for the Strong Gap

## TL;DR

This paper develops differentially private algorithms for stochastic saddle point problems, achieving near-optimal convergence rates and analyzing the tradeoff between stability and accuracy in such settings.

## Contribution

It introduces a novel recursive regularization technique for saddle point problems under differential privacy constraints, achieving optimal rates and providing a general algorithm framework.

## Key findings

- Achieves nearly optimal strong gap rates of rac{1}{\u221a{n}} + rac{\u221a{d}}{npsilon}
- Develops a general algorithm with rac{ ext{min}igrac{n^2\u2215 ext{epsilon}^{1.5}}{\u221a{d}}, n^{3/2}ig)} gradient complexity
- Establishes a fundamental tradeoff between stability and accuracy in differentially private algorithms.

## Abstract

We show that convex-concave Lipschitz stochastic saddle point problems (also known as stochastic minimax optimization) can be solved under the constraint of $(\epsilon,\delta)$-differential privacy with \emph{strong (primal-dual) gap} rate of $\tilde O\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$, where $n$ is the dataset size and $d$ is the dimension of the problem. This rate is nearly optimal, based on existing lower bounds in differentially private stochastic optimization. Specifically, we prove a tight upper bound on the strong gap via novel implementation and analysis of the recursive regularization technique repurposed for saddle point problems. We show that this rate can be attained with $O\big(\min\big\{\frac{n^2\epsilon^{1.5}}{\sqrt{d}}, n^{3/2}\big\}\big)$ gradient complexity, and $\tilde{O}(n)$ gradient complexity if the loss function is smooth. As a byproduct of our method, we develop a general algorithm that, given a black-box access to a subroutine satisfying a certain $\alpha$ primal-dual accuracy guarantee with respect to the empirical objective, gives a solution to the stochastic saddle point problem with a strong gap of $\tilde{O}(\alpha+\frac{1}{\sqrt{n}})$. We show that this $\alpha$-accuracy condition is satisfied by standard algorithms for the empirical saddle point problem such as the proximal point method and the stochastic gradient descent ascent algorithm. Further, we show that even for simple problems it is possible for an algorithm to have zero weak gap and suffer from $\Omega(1)$ strong gap. We also show that there exists a fundamental tradeoff between stability and accuracy. Specifically, we show that any $\Delta$-stable algorithm has empirical gap $\Omega\big(\frac{1}{\Delta n}\big)$, and that this bound is tight. This result also holds also more specifically for empirical risk minimization problems and may be of independent interest.