An Efficient Stochastic First-Order Algorithm for Nonconvex-Strongly Concave Minimax Optimization beyond Lipschitz Smoothness

TL;DR

This paper introduces a new stochastic first-order algorithm, NSGDA-M, for nonconvex-strongly concave minimax problems under generalized smoothness, achieving improved convergence guarantees beyond traditional Lipschitz smoothness assumptions.

Contribution

The paper proposes NSGDA-M, a novel algorithm for stochastic minimax problems under generalized smoothness, with proven convergence rates for nonconvex-strongly concave cases.

Findings

NSGDA-M finds an $ ext{epsilon}$-stationary point within $ ext{O}( ext{epsilon}^{-4})$ gradient evaluations.

The algorithm achieves high probability convergence with $ ext{O}( ext{epsilon}^{-4}( ext{log}(1/ ext{delta}))^{3/2})$ evaluations.

Numerical experiments demonstrate the effectiveness of NSGDA-M on distributionally robust optimization tasks.

Abstract

In recent years, nonconvex minimax problems have attracted significant attention due to their broad applications in machine learning, including generative adversarial networks, robust optimization and adversarial training. Most existing algorithms for nonconvex stochastic minimax problems are developed under the standard Lipschitz smoothness assumption. In this paper, we study stochastic minimax problems under a generalized smoothness condition and propose an algorithm, NSGDA-M, which simultaneously updates the inner variable by stochastic gradient ascent and updates the outer variable by normalized stochastic gradient descent with momentum. When the objective function is nonconvex-strongly concave, we show that NSGDA-M finds an $\epsilon$-stationary point of the primal function within $\mathcal{O}(\epsilon^{-4})$ stochastic gradient evaluations in expectation, and $\mathcal{O}\left(\epsilon^{-4}(\log(\frac{1}{\delta}))^{3/2}\right)$ stochastic gradient evaluations in high probability, where $\delta\in (0,1)$ is the failure probability. We verify the effectiveness of the proposed algorithm through numerical experiments on a distributionally robust optimization problem.