A Universal Algorithm for Variational Inequalities Adaptive to Smoothness and Noise

TL;DR

This paper introduces a universal, adaptive algorithm for variational inequalities that automatically adjusts to smoothness and noise levels, achieving optimal convergence without prior knowledge of problem properties.

Contribution

The authors develop a universal Mirror-Prox based algorithm that adapts to smoothness and noise, extending AdaGrad to constrained variational inequality problems.

Findings

Achieves optimal rates in smooth/non-smooth and noisy/noiseless settings

Works without prior knowledge of problem smoothness or noise levels

Applies to convex minimization and saddle-point problems

Abstract

We consider variational inequalities coming from monotone operators, a setting that includes convex minimization and convex-concave saddle-point problems. We assume an access to potentially noisy unbiased values of the monotone operators and assess convergence through a compatible gap function which corresponds to the standard optimality criteria in the aforementioned subcases. We present a universal algorithm for these inequalities based on the Mirror-Prox algorithm. Concretely, our algorithm simultaneously achieves the optimal rates for the smooth/non-smooth, and noisy/noiseless settings. This is done without any prior knowledge of these properties, and in the general set-up of arbitrary norms and compatible Bregman divergences. For convex minimization and convex-concave saddle-point problems, this leads to new adaptive algorithms. Our method relies on a novel yet simple adaptive choice of the step-size, which can be seen as the appropriate extension of AdaGrad to handle constrained problems.