Two-level preconditioning for Ridge Regression

TL;DR

This paper introduces a two-level preconditioning method for solving regularized least squares problems like ridge regression, significantly speeding up iterative solvers by using clustering-based coarser levels to approximate dominant eigenvectors.

Contribution

The paper presents a novel two-level preconditioner for regularized least squares systems, leveraging clustering to create a coarser level that enhances convergence of iterative solvers.

Findings

Speed-ups observed on artificial data

Speed-ups observed on real-life data

Effective approximation of dominant eigenvectors

Abstract

Solving linear systems is often the computational bottleneck in real-life problems. Iterative solvers are the only option due to the complexity of direct algorithms or because the system matrix is not explicitly known. Here, we develop a two-level preconditioner for regularized least squares linear systems involving a feature or data matrix. Variants of this linear system may appear in machine learning applications, such as ridge regression, logistic regression, support vector machines and Bayesian regression. We use clustering algorithms to create a coarser level that preserves the principal components of the covariance or Gram matrix. This coarser level approximates the dominant eigenvectors and is used to build a subspace preconditioner accelerating the Conjugate Gradient method. We observed speed-ups for artificial and real-life data.