Randomized Block-Coordinate Optimistic Gradient Algorithms for Root-Finding Problems

TL;DR

This paper introduces two novel randomized block-coordinate optimistic gradient algorithms for solving large-scale root-finding problems, achieving improved convergence rates and applicability to federated learning scenarios.

Contribution

The paper presents the first accelerated randomized block-coordinate optimistic gradient algorithm with $ ext{O}(1/k^2)$ convergence and applies these methods to large-scale finite-sum inclusions in machine learning.

Findings

Achieved $ ext{O}(1/k)$ convergence rate for the non-accelerated method.

Established $ ext{O}(1/k^2)$ and $o(1/k^2)$ convergence rates for the accelerated method.

Developed federated learning algorithms with proven convergence guarantees.

Abstract

In this paper, we develop two new randomized block-coordinate optimistic gradient algorithms to approximate a solution of nonlinear equations in large-scale settings, which are called root-finding problems. Our first algorithm is non-accelerated with constant stepsizes, and achieves $\mathcal{O}(1/k)$ best-iterate convergence rate on $\mathbb{E}[ \Vert Gx^k\Vert^2]$ when the underlying operator $G$ is Lipschitz continuous and satisfies a weak Minty solution condition, where $\mathbb{E}[\cdot]$ is the expectation and $k$ is the iteration counter. Our second method is a new accelerated randomized block-coordinate optimistic gradient algorithm. We establish both $\mathcal{O}(1/k^2)$ and $o(1/k^2)$ last-iterate convergence rates on both $\mathbb{E}[ \Vert Gx^k\Vert^2]$ and $\mathbb{E}[ \Vert x^{k+1} - x^{k}\Vert^2]$ for this algorithm under the co-coerciveness of $G$. In addition, we prove that the iterate sequence $\{x^k\}$ converges to a solution almost surely, and $k\Vert Gx^k\Vert$ attains a $o(1/k)$ almost sure convergence rate. Then, we apply our methods to a class of large-scale finite-sum inclusions, which covers prominent applications in machine learning, statistical learning, and network optimization, especially in federated learning. We obtain two new federated learning-type algorithms and their convergence rate guarantees for solving this problem class.