A Divide-and-Conquer Solver for Kernel Support Vector Machines

TL;DR

This paper introduces a divide-and-conquer approach for kernel SVMs that significantly speeds up training on large datasets while maintaining high accuracy, by partitioning data and efficiently combining solutions.

Contribution

The paper proposes a novel multilevel divide-and-conquer algorithm for kernel SVMs, with theoretical support and practical improvements over existing methods.

Findings

DC-SVM is 7 times faster than LIBSVM on large datasets.

The method achieves 96.15% accuracy with much less training time.

Early prediction strategy reduces training time by over 100 times.

Abstract

The kernel support vector machine (SVM) is one of the most widely used classification methods; however, the amount of computation required becomes the bottleneck when facing millions of samples. In this paper, we propose and analyze a novel divide-and-conquer solver for kernel SVMs (DC-SVM). In the division step, we partition the kernel SVM problem into smaller subproblems by clustering the data, so that each subproblem can be solved independently and efficiently. We show theoretically that the support vectors identified by the subproblem solution are likely to be support vectors of the entire kernel SVM problem, provided that the problem is partitioned appropriately by kernel clustering. In the conquer step, the local solutions from the subproblems are used to initialize a global coordinate descent solver, which converges quickly as suggested by our analysis. By extending this idea, we develop a multilevel Divide-and-Conquer SVM algorithm with adaptive clustering and early prediction strategy, which outperforms state-of-the-art methods in terms of training speed, testing accuracy, and memory usage. As an example, on the covtype dataset with half-a-million samples, DC-SVM is 7 times faster than LIBSVM in obtaining the exact SVM solution (to within $10^{-6}$ relative error) which achieves 96.15% prediction accuracy. Moreover, with our proposed early prediction strategy, DC-SVM achieves about 96% accuracy in only 12 minutes, which is more than 100 times faster than LIBSVM.