Breaking the Stochasticity Barrier: An Adaptive Variance-Reduced Method for Variational Inequalities

TL;DR

This paper introduces VR-SDA-A, an adaptive variance-reduced method for stochastic variational inequalities, achieving optimal complexity and automatic step-size adaptation, overcoming the stochasticity barrier in non-convex saddle-point problems.

Contribution

The paper proposes VR-SDA-A, a novel algorithm combining recursive momentum and curvature verification, enabling adaptive step-size in stochastic variational inequalities with proven optimal complexity.

Findings

Achieves O(epsilon^{-3}) oracle complexity for stationary points.

Effectively suppresses limit cycles in rotational benchmarks.

Accelerates convergence in non-convex robust regression tasks.

Abstract

Stochastic non-convex non-concave optimization, formally characterized as Stochastic Variational Inequalities (SVIs), presents unique challenges due to rotational dynamics and the absence of a global merit function. While adaptive step-size methods (like Armijo line-search) have revolutionized convex minimization, their application to this setting is hindered by the Stochasticity Barrier: the noise in gradient estimation masks the true operator curvature, triggering erroneously large steps that destabilize convergence. In this work, we propose VR-SDA-A (Variance-Reduced Stochastic Descent-Ascent with Armijo), a novel algorithm that integrates recursive momentum (STORM) with a rigorous Same-Batch Curvature Verification mechanism. We introduce a theoretical framework based on a Lyapunov potential tracking the Operator Norm, proving that VR- SDA-A achieves an oracle complexity of O(epsilon -3) for finding an epsilon-stationary point in general Lipschitz continuous operators. This matches the optimal rate for non-convex minimization while uniquely enabling automated step-size adaptation in the saddle-point setting. We validate our approach on canonical rotational benchmarks and non-convex robust regression tasks, demonstrating that our method effectively suppresses limit cycles and accelerates convergence with reduced dependence on manual learning rate scheduling.