Large Scale Kernel Learning using Block Coordinate Descent

TL;DR

This paper shows that distributed block coordinate descent efficiently solves large-scale kernel learning problems and compares different approximation methods, providing new theoretical insights and practical results.

Contribution

It introduces a distributed block coordinate descent approach for large-scale kernel learning and derives new convergence rates specific to kernel methods.

Findings

Nyström method generally outperforms random features in accuracy

Nyström method requires more optimization iterations

New convergence rates support experimental results

Abstract

We demonstrate that distributed block coordinate descent can quickly solve kernel regression and classification problems with millions of data points. Armed with this capability, we conduct a thorough comparison between the full kernel, the Nystr\"om method, and random features on three large classification tasks from various domains. Our results suggest that the Nystr\"om method generally achieves better statistical accuracy than random features, but can require significantly more iterations of optimization. Lastly, we derive new rates for block coordinate descent which support our experimental findings when specialized to kernel methods.