A Unified Analysis of Variational Inequality Methods: Variance Reduction, Sampling, Quantization and Coordinate Descent

TL;DR

This paper provides a comprehensive analysis of variational inequality methods, unifying and extending existing algorithms with new robust techniques applicable to a broad class of problems, including saddle point and minimization problems.

Contribution

It introduces a unified theoretical framework for variational inequality methods and develops new robust algorithms with practical applications, such as GAN training.

Findings

New algorithms with quantization and coordinate methods

Theoretical foundation for combining existing variational methods

Numerical validation on GANs

Abstract

In this paper, we present a unified analysis of methods for such a wide class of problems as variational inequalities, which includes minimization problems and saddle point problems. We develop our analysis on the modified Extra-Gradient method (the classic algorithm for variational inequalities) and consider the strongly monotone and monotone cases, which corresponds to strongly-convex-strongly-concave and convex-concave saddle point problems. The theoretical analysis is based on parametric assumptions about Extra-Gradient iterations. Therefore, it can serve as a strong basis for combining the already existing type methods and also for creating new algorithms. In particular, to show this we develop new robust methods, which include methods with quantization, coordinate methods, distributed randomized local methods, and others. Most of these approaches have never been considered in the generality of variational inequalities and have previously been used only for minimization problems. The robustness of the new methods is also confirmed by numerical experiments with GANs.