The Complexity of Primal-Dual Fixed Point Methods for Ridge Regression

TL;DR

This paper analyzes the complexity of primal-dual fixed point methods for ridge regression, introduces a relaxation parameter, and compares various reformulations, revealing connections between spectral properties and convergence rates, with practical implications.

Contribution

It introduces a structured framework for analyzing fixed point methods for ridge regression, including optimal relaxation parameters and spectral analysis, and demonstrates the effectiveness of the Quartz method.

Findings

The Quartz method achieves the best theoretical and numerical convergence rates.

Spectral properties of reformulated systems influence fixed point method performance.

Numerical experiments show competitiveness with conjugate gradient methods.

Abstract

We study the ridge regression (L2 regularized least squares) problem and its dual, which is also a ridge regression problem. We observe that the optimality conditions describing the primal and dual optimal solutions can be formulated in several different but equivalent ways. The optimality conditions we identify form a linear system involving a structured matrix depending on a single relaxation parameter which we introduce for regularization purposes. This leads to the idea of studying and comparing, in theory and practice, the performance of the fixed point method applied to these reformulations. We compute the optimal relaxation parameters and uncover interesting connections between the complexity bounds of the variants of the fixed point scheme we consider. These connections follow from a close link between the spectral properties of the associated matrices. For instance, some reformulations involve purely imaginary eigenvalues; some involve real eigenvalues and others have all eigenvalues on the complex circle. We show that the deterministic Quartz method---which is a special case of the randomized dual coordinate ascent method with arbitrary sampling recently developed by Qu, Richt\'{a}rik and Zhang---can be cast in our framework, and achieves the best rate in theory and in numerical experiments among the fixed point methods we study. Remarkably, the method achieves an accelerated convergence rate. Numerical experiments indicate that our main algorithm is competitive with the conjugate gradient method.