A Generalized Matrix Splitting Algorithm

TL;DR

The paper introduces a Generalized Matrix Splitting Algorithm (GMSA) for efficiently minimizing composite functions, with proven convergence and superior performance in applications like sparse coding and matrix factorization.

Contribution

It presents a novel GMSA based on a triangle operator and generalized Gaussian elimination, extending classical methods for broader composite function minimization.

Findings

Proven global convergence and convergence rate of GMSA.

State-of-the-art performance in efficiency and efficacy.

Successful application to nonnegative matrix factorization, sparse coding, and Dantzig selector.

Abstract

Composite function minimization captures a wide spectrum of applications in both computer vision and machine learning. It includes bound constrained optimization, $\ell_1$ norm regularized optimization, and $\ell_0$ norm regularized optimization as special cases. This paper proposes and analyzes a new Generalized Matrix Splitting Algorithm (GMSA) for minimizing composite functions. It can be viewed as a generalization of the classical Gauss-Seidel method and the Successive Over-Relaxation method for solving linear systems in the literature. Our algorithm is derived from a novel triangle operator mapping, which can be computed exactly using a new generalized Gaussian elimination procedure. We establish the global convergence, convergence rate, and iteration complexity of GMSA for convex problems. In addition, we also discuss several important extensions of GMSA. Finally, we validate the performance of our proposed method on three particular applications: nonnegative matrix factorization, $\ell_0$ norm regularized sparse coding, and $\ell_1$ norm regularized Dantzig selector problem. Extensive experiments show that our method achieves state-of-the-art performance in term of both efficiency and efficacy.