Revisiting Extragradient-Type Methods -- Part 1: Generalizations and Sublinear Convergence Rates

TL;DR

This paper provides a unified framework for extragradient methods, extending their applicability to a broader class of problems, analyzing convergence rates, and introducing new variants with improved performance validated by experiments.

Contribution

It generalizes extragradient methods for equations and inclusions, introduces new algorithms, and establishes convergence rates, broadening the scope of existing methods.

Findings

New variants outperform existing schemes in most examples.

Sublinear convergence rates are established for various algorithms.

Unified analysis encompasses multiple extragradient-based methods.

Abstract

This paper presents a comprehensive analysis of the well-known extragradient (EG) method for solving both equations and inclusions. First, we unify and generalize EG for [non]linear equations to a wider class of algorithms, encompassing various existing schemes and potentially new variants. Next, we analyze both sublinear ``best-iterate'' and ``last-iterate'' convergence rates for the entire class of algorithms, and derive new convergence results for two well-known instances. Second, we extend our EG framework above to ``monotone'' inclusions, introducing a new class of algorithms and its corresponding convergence results. Third, we also unify and generalize Tseng's forward-backward-forward splitting (FBFS) method to a broader class of algorithms to solve [non]linear inclusions when a weak-Minty solution exists, and establish its ``best-iterate'' convergence rate. Fourth, to complete our picture, we also investigate sublinear rates of two other common variants of EG using our EG analysis framework developed here: the reflected forward-backward splitting and the golden ratio methods. Finally, we conduct an extensive numerical experiment to validate our theoretical findings. Our results demonstrate that several new variants of our proposed algorithms outperform existing schemes in the majority of examples.