Exploring Jacobian Inexactness in Second-Order Methods for Variational Inequalities: Lower Bounds, Optimal Algorithms and Quasi-Newton Approximations

TL;DR

This paper investigates how Jacobian inaccuracy affects second-order methods for variational inequalities, establishing lower bounds, proposing optimal algorithms, and introducing Quasi-Newton approximations to improve efficiency.

Contribution

It provides the first lower bounds for Jacobian inaccuracy impact, develops an optimal second-order algorithm, and introduces Quasi-Newton methods for variational inequalities.

Findings

Lower bounds depend explicitly on Jacobian inaccuracy.

Proposed algorithms match exact derivative convergence rates.

Quasi-Newton methods achieve global sublinear convergence.

Abstract

Variational inequalities represent a broad class of problems, including minimization and min-max problems, commonly found in machine learning. Existing second-order and high-order methods for variational inequalities require precise computation of derivatives, often resulting in prohibitively high iteration costs. In this work, we study the impact of Jacobian inaccuracy on second-order methods. For the smooth and monotone case, we establish a lower bound with explicit dependence on the level of Jacobian inaccuracy and propose an optimal algorithm for this key setting. When derivatives are exact, our method converges at the same rate as exact optimal second-order methods. To reduce the cost of solving the auxiliary problem, which arises in all high-order methods with global convergence, we introduce several Quasi-Newton approximations. Our method with Quasi-Newton updates achieves a global sublinear convergence rate. We extend our approach with a tensor generalization for inexact high-order derivatives and support the theory with experiments.