TiAda: A Time-scale Adaptive Algorithm for Nonconvex Minimax Optimization

TL;DR

TiAda is a novel adaptive gradient algorithm designed for nonconvex minimax problems, automatically adjusting to time-scale separation and achieving near-optimal convergence without hyper-parameter tuning.

Contribution

The paper introduces TiAda, a parameter-agnostic, single-loop adaptive GDA algorithm that handles nonconvex minimax optimization with automatic time-scale adaptation.

Findings

Achieves near-optimal convergence complexities in deterministic and stochastic settings.

Automatically adapts to primal-dual time-scale separation.

Demonstrates effectiveness in machine learning applications.

Abstract

Adaptive gradient methods have shown their ability to adjust the stepsizes on the fly in a parameter-agnostic manner, and empirically achieve faster convergence for solving minimization problems. When it comes to nonconvex minimax optimization, however, current convergence analyses of gradient descent ascent (GDA) combined with adaptive stepsizes require careful tuning of hyper-parameters and the knowledge of problem-dependent parameters. Such a discrepancy arises from the primal-dual nature of minimax problems and the necessity of delicate time-scale separation between the primal and dual updates in attaining convergence. In this work, we propose a single-loop adaptive GDA algorithm called TiAda for nonconvex minimax optimization that automatically adapts to the time-scale separation. Our algorithm is fully parameter-agnostic and can achieve near-optimal complexities simultaneously in deterministic and stochastic settings of nonconvex-strongly-concave minimax problems. The effectiveness of the proposed method is further justified numerically for a number of machine learning applications.