A Linearly Convergent Doubly Stochastic Gauss-Seidel Algorithm for Solving Linear Equations and A Certain Class of Over-Parameterized Optimization Problems

TL;DR

This paper introduces a doubly stochastic Gauss-Seidel algorithm that guarantees linear convergence for any linear system, overcoming traditional limitations of divergence in non-diagonally dominant matrices, and extends to related optimization problems.

Contribution

It proposes a novel nonuniform double stochastic scheme with a specific stepsize rule, ensuring linear convergence for all feasible linear systems and related optimization tasks.

Findings

Proves linear convergence of the proposed algorithm for any feasible linear system.

Generalizes the approach to linear feasibility problems and high-dimensional machine learning models.

Demonstrates that randomization can ensure convergence where classical methods may diverge.

Abstract

Consider the classical problem of solving a general linear system of equations $Ax=b$. It is well known that the (successively over relaxed) Gauss-Seidel scheme and many of its variants may not converge when $A$ is neither diagonally dominant nor symmetric positive definite. Can we have a linearly convergent G-S type algorithm that works for {\it any} $A$? In this paper we answer this question affirmatively by proposing a doubly stochastic G-S algorithm that is provably linearly convergent (in the mean square error sense) for any feasible linear system of equations. The key in the algorithm design is to introduce a {\it nonuniform double stochastic} scheme for picking the equation and the variable in each update step as well as a stepsize rule. These techniques also generalize to certain iterative alternating projection algorithms for solving the linear feasibility problem $A x\le b$ with an arbitrary $A$, as well as high-dimensional minimization problems for training over-parameterized models in machine learning. Our results demonstrate that a carefully designed randomization scheme can make an otherwise divergent G-S algorithm converge.