Accelerated Proximal Alternating Gradient-Descent-Ascent for Nonconvex Minimax Machine Learning

TL;DR

This paper introduces a fast, single-loop proximal gradient algorithm with momentum acceleration for nonconvex minimax problems, improving convergence efficiency over existing methods and demonstrating effectiveness in adversarial deep learning.

Contribution

It develops a novel accelerated AltGDA-type algorithm with proven convergence and better complexity bounds for nonconvex minimax optimization.

Findings

Achieves convergence to critical points in nonconvex minimax problems.

Reduces computational complexity compared to existing AltGDA algorithms.

Demonstrates effectiveness in adversarial deep learning experiments.

Abstract

Alternating gradient-descent-ascent (AltGDA) is an optimization algorithm that has been widely used for model training in various machine learning applications, which aims to solve a nonconvex minimax optimization problem. However, the existing studies show that it suffers from a high computation complexity in nonconvex minimax optimization. In this paper, we develop a single-loop and fast AltGDA-type algorithm that leverages proximal gradient updates and momentum acceleration to solve regularized nonconvex minimax optimization problems. By leveraging the momentum acceleration technique, we prove that the algorithm converges to a critical point in nonconvex minimax optimization and achieves a computation complexity in the order of $\mathcal{O}(\kappa^{\frac{11}{6}}\epsilon^{-2})$, where $\epsilon$ is the desired level of accuracy and $\kappa$ is the problem's condition number. {Such a computation complexity improves the state-of-the-art complexities of single-loop GDA and AltGDA algorithms (see the summary of comparison in \Cref{table1})}. We demonstrate the effectiveness of our algorithm via an experiment on adversarial deep learning.