Gradient-free algorithm for saddle point problems under overparametrization

TL;DR

This paper introduces a gradient-free method for stochastic saddle point problems in overparameterized settings, providing convergence analysis and noise robustness insights.

Contribution

It generalizes the Stochastic Extra-gradient algorithm to biased oracles and develops a gradient approximation approach for saddle point problems.

Findings

Derived convergence rates for the biased oracle setting

Identified maximum noise levels for guaranteed accuracy

Determined optimal iteration counts for convergence

Abstract

This paper focuses on solving a stochastic saddle point problem (SPP) under an overparameterized regime for the case, when the gradient computation is impractical. As an intermediate step, we generalize Same-sample Stochastic Extra-gradient algorithm (Gorbunov et al., 2022) to a biased oracle and estimate novel convergence rates. As the result of the paper we introduce an algorithm, which uses gradient approximation instead of a gradient oracle. We also conduct an analysis to find the maximum admissible level of adversarial noise and the optimal number of iterations at which our algorithm can guarantee achieving the desired accuracy.