Fixed-point and coordinate descent algorithms for regularized kernel methods

TL;DR

This paper introduces fixed-point and coordinate descent algorithms for large-scale kernel methods with convex loss functions, providing convergence analysis and practical implementation insights.

Contribution

It presents two general classes of algorithms, fixed-point and coordinate descent, tailored for kernel methods with convex loss and quadratic regularization, including convergence analysis.

Findings

Algorithms are simple to implement and parallelize.

Solutions can be characterized using sub-differential calculus.

Applicable to large-scale problems with convex loss functions.

Abstract

In this paper, we study two general classes of optimization algorithms for kernel methods with convex loss function and quadratic norm regularization, and analyze their convergence. The first approach, based on fixed-point iterations, is simple to implement and analyze, and can be easily parallelized. The second, based on coordinate descent, exploits the structure of additively separable loss functions to compute solutions of line searches in closed form. Instances of these general classes of algorithms are already incorporated into state of the art machine learning software for large scale problems. We start from a solution characterization of the regularized problem, obtained using sub-differential calculus and resolvents of monotone operators, that holds for general convex loss functions regardless of differentiability. The two methodologies described in the paper can be regarded as instances of non-linear Jacobi and Gauss-Seidel algorithms, and are both well-suited to solve large scale problems.