Zeroth-Order Algorithms for Smooth Saddle-Point Problems

TL;DR

This paper introduces zeroth-order algorithms for smooth saddle-point problems, including stochastic and mixed methods, with theoretical convergence analysis and practical performance evaluation.

Contribution

It develops new zeroth-order and 1/2th-order algorithms for saddle-point problems, analyzing their convergence rates and demonstrating their effectiveness in practice.

Findings

Convergence rate for zeroth-order methods is only a log n factor worse than first-order methods on simplices.

Proposed algorithms perform well on practical saddle-point problems.

Mixed 1/2th-order methods effectively combine zeroth- and first-order oracles.

Abstract

Saddle-point problems have recently gained increased attention from the machine learning community, mainly due to applications in training Generative Adversarial Networks using stochastic gradients. At the same time, in some applications only a zeroth-order oracle is available. In this paper, we propose several algorithms to solve stochastic smooth (strongly) convex-concave saddle-point problems using zeroth-order oracles and estimate their convergence rate and its dependence on the dimension $n$ of the variable. In particular, our analysis shows that in the case when the feasible set is a direct product of two simplices, our convergence rate for the stochastic term is only by a $\log n$ factor worse than for the first-order methods. We also consider a mixed setup and develop 1/2th-order methods that use zeroth-order oracle for the minimization part and first-order oracle for the maximization part. Finally, we demonstrate the practical performance of our zeroth-order and 1/2th-order methods on practical problems.