Recipe for Fast Large-scale SVM Training: Polishing, Parallelism, and more RAM!

TL;DR

This paper presents a novel approach combining approximation and parallelism to significantly accelerate large-scale SVM training on modern hardware, enabling training on massive datasets like ImageNet in under half an hour.

Contribution

It introduces a fast dual SVM solver that leverages multi-core CPUs, multiple GPUs, and large RAM, achieving unprecedented training speeds for large datasets.

Findings

Training on ImageNet in 24 minutes

Effective utilization of modern compute hardware

Significant speedup over traditional methods

Abstract

Support vector machines (SVMs) are a standard method in the machine learning toolbox, in particular for tabular data. Non-linear kernel SVMs often deliver highly accurate predictors, however, at the cost of long training times. That problem is aggravated by the exponential growth of data volumes over time. It was tackled in the past mainly by two types of techniques: approximate solvers, and parallel GPU implementations. In this work, we combine both approaches to design an extremely fast dual SVM solver. We fully exploit the capabilities of modern compute servers: many-core architectures, multiple high-end GPUs, and large random access memory. On such a machine, we train a large-margin classifier on the ImageNet data set in 24 minutes.