Regularized Operator Extrapolation Method For Stochastic Bilevel Variational Inequality Problems

TL;DR

This paper introduces R-OpEx, a novel first-order method for stochastic bilevel variational inequality problems that achieves improved complexity bounds and handles nonsmooth, stochastic, and deterministic cases with strong theoretical guarantees.

Contribution

The paper presents R-OpEx, a single-loop regularized operator extrapolation method for BVI, providing the first convergence guarantees for nonsmooth stochastic BVI and improved complexities.

Findings

Achieves $ ext{O}( ext{epsilon}^{-4})$ complexity for nonsmooth stochastic BVI.

Improves outer level complexity to $ ext{O}( ext{epsilon}^{-2})$ with mini-batching.

Attains $ ext{O}( ext{epsilon}^{-4/5})$ complexity for strongly monotone outer levels.

Abstract

The bilevel variational inequality (BVI) problem is a general model that captures various optimization problems, including VI-constrained optimization and equilibrium problems with equilibrium constraints (EPECs). This paper introduces a first-order method for smooth or nonsmooth BVI with stochastic monotone operators at inner and outer levels. Our novel method, called Regularized Operator Extrapolation $(\texttt{R-OpEx})$, is a single-loop algorithm that combines Tikhonov's regularization with operator extrapolation. This method needs only one operator evaluation for each operator per iteration and tracks one sequence of iterates. We show that $\texttt{R-OpEx}$ gives $\mathcal{O}(\epsilon^{-4})$ complexity in nonsmooth stochastic monotone BVI, where $\epsilon$ is the error in the inner and outer levels. Using a mini-batching scheme, we improve the outer level complexity to $\mathcal{O}(\epsilon^{-2})$ while maintaining the $\mathcal{O}(\epsilon^{-4})$ complexity in the inner level when the inner level is smooth and stochastic. Moreover, if the inner level is smooth and deterministic, we show complexity of $\mathcal{O}(\epsilon^{-2})$. Finally, in case the outer level is strongly monotone, we improve to $\mathcal{O}(\epsilon^{-4/5})$ for general BVI and $\mathcal{O}(\epsilon^{-2/3})$ when the inner level is smooth and deterministic. To our knowledge, this is the first work that investigates nonsmooth stochastic BVI with the best-known convergence guarantees. We verify our theoretical results with numerical experiments.