General Procedure to Provide High-Probability Guarantees for Stochastic Saddle Point Problems

TL;DR

This paper introduces a general high-probability framework for stochastic saddle point problems that guarantees small duality gaps with fewer oracle calls, improving sample complexity and reliability.

Contribution

It proposes a novel PB-SSP framework that provides high-probability guarantees for stochastic saddle point solutions with reduced oracle complexity and enhanced sample efficiency.

Findings

Guarantees small duality gap with high probability using logarithmic oracle calls.

Improves sample complexity of the SAA oracle by a polynomial factor.

Circumvents stability analysis difficulties through high-probability arguments.

Abstract

This paper considers smooth strongly convex and strongly concave (SC-SC) stochastic saddle point (SSP) problems. Suppose there is an arbitrary oracle that in expectation returns an $\epsilon$-solution in the sense of certain gaps, which can be the duality gap or its weaker variants. We propose a general PB-SSP framework to guarantee an $\epsilon$ small duality gap solution with high probability via only $\mathcal{O}\big(\log \frac{1}{p}\cdot\text{poly}(\log \kappa)\big)$ calls of this oracle, where $p\in(0,1)$ is the confidence level and $\kappa$ is the condition number. When applied to the sample average approximation (SAA) oracle, in addition to equipping the solution with high probability, our approach even improves the sample complexity by a factor of $\text{poly}(\kappa)$, since the high-probability argument enables us to circumvent some key difficulties of the uniform stability analysis of SAA.