Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentum

TL;DR

This paper introduces bias-corrected momentum algorithms for nonconvex-strongly-concave minimax problems, achieving improved convergence rates of ()) and validated on real-world robust logistic regression tasks.

Contribution

It proposes novel bias-corrected momentum algorithms that leverage Hessian-vector products to improve convergence rates in nonconvex-strongly-concave minimax optimization.

Findings

Achieves ()) iteration complexity for convergence.

Demonstrates effectiveness on real-world robust logistic regression datasets.

Provides convergence analysis under specific conditions.

Abstract

Lower-bound analyses for nonconvex strongly-concave minimax optimization problems have shown that stochastic first-order algorithms require at least $\mathcal{O}(\varepsilon^{-4})$ oracle complexity to find an $\varepsilon$-stationary point. Some works indicate that this complexity can be improved to $\mathcal{O}(\varepsilon^{-3})$ when the loss gradient is Lipschitz continuous. The question of achieving enhanced convergence rates under distinct conditions, remains unresolved. In this work, we address this question for optimization problems that are nonconvex in the minimization variable and strongly concave or Polyak-Lojasiewicz (PL) in the maximization variable. We introduce novel bias-corrected momentum algorithms utilizing efficient Hessian-vector products. We establish convergence conditions and demonstrate a lower iteration complexity of $\mathcal{O}(\varepsilon^{-3})$ for the proposed algorithms. The effectiveness of the method is validated through applications to robust logistic regression using real-world datasets.