Scalable Dual Coordinate Descent for Kernel Methods

TL;DR

This paper introduces $s$-step variants of dual coordinate descent methods for kernel SVMs and ridge regression, significantly reducing communication costs and achieving near-linear speedups on large-scale distributed systems.

Contribution

The paper develops and analyzes $s$-step variants of DCD and BDCD methods that reduce communication frequency, improving scalability for kernel methods on distributed hardware.

Findings

Achieved up to 9.8x speedup with 512 cores.

Maintained numerical stability for large $s$ values.

Bounded computation and communication costs theoretically.

Abstract

Dual Coordinate Descent (DCD) and Block Dual Coordinate Descent (BDCD) are important iterative methods for solving convex optimization problems. In this work, we develop scalable DCD and BDCD methods for the kernel support vector machines (K-SVM) and kernel ridge regression (K-RR) problems. On distributed-memory parallel machines the scalability of these methods is limited by the need to communicate every iteration. On modern hardware where communication is orders of magnitude more expensive, the running time of the DCD and BDCD methods is dominated by communication cost. We address this communication bottleneck by deriving $s$-step variants of DCD and BDCD for solving the K-SVM and K-RR problems, respectively. The $s$-step variants reduce the frequency of communication by a tunable factor of $s$ at the expense of additional bandwidth and computation. The $s$-step variants compute the same solution as the existing methods in exact arithmetic. We perform numerical experiments to illustrate that the $s$-step variants are also numerically stable in finite-arithmetic, even for large values of $s$. We perform theoretical analysis to bound the computation and communication costs of the newly designed variants, up to leading order. Finally, we develop high performance implementations written in C and MPI and present scaling experiments performed on a Cray EX cluster. The new $s$-step variants achieved strong scaling speedups of up to $9.8\times$ over existing methods using up to $512$ cores.