Beyond the Golden Ratio for Variational Inequality Algorithms

TL;DR

This paper advances the understanding of adaptive algorithms for variational inequalities and min-max problems, removing the need for hyperparameters and global Lipschitz assumptions, and extends their applicability to more general settings.

Contribution

It establishes the equivalence of the golden ratio algorithm with popular VI methods, introduces a new analysis for constrained problems, and improves the adaptive algorithm for nonmonotone cases.

Findings

Eliminates the link between the golden ratio and the algorithm.

Improves the adaptive algorithm's complexity bounds.

Enhances empirical performance on nonmonotone problems.

Abstract

We improve the understanding of the $\textit{golden ratio algorithm}$, which solves monotone variational inequalities (VI) and convex-concave min-max problems via the distinctive feature of adapting the step sizes to the local Lipschitz constants. Adaptive step sizes not only eliminate the need to pick hyperparameters, but they also remove the necessity of global Lipschitz continuity and can increase from one iteration to the next. We first establish the equivalence of this algorithm with popular VI methods such as reflected gradient, Popov or optimistic gradient descent-ascent in the unconstrained case with constant step sizes. We then move on to the constrained setting and introduce a new analysis that allows to use larger step sizes, to complete the bridge between the golden ratio algorithm and the existing algorithms in the literature. Doing so, we actually eliminate the link between the golden ratio $\frac{1+\sqrt{5}}{2}$ and the algorithm. Moreover, we improve the adaptive version of the algorithm, first by removing the maximum step size hyperparameter (an artifact from the analysis) to improve the complexity bound, and second by adjusting it to nonmonotone problems with weak Minty solutions, with superior empirical performance.