Accelerated Linear SVM Training with Adaptive Variable Selection Frequencies

TL;DR

This paper introduces an adaptive variable selection frequency method to accelerate linear SVM training, significantly improving training speed over existing algorithms by dynamically adjusting selection strategies.

Contribution

It proposes an online adaptive selection frequency mechanism that enhances the efficiency of the liblinear algorithm for linear SVM training.

Findings

Speeds up training by more than an order of magnitude in some cases

Replaces uniform selection with adaptive, data-driven strategies

Improves upon existing dual decomposition algorithms for linear SVMs

Abstract

Support vector machine (SVM) training is an active research area since the dawn of the method. In recent years there has been increasing interest in specialized solvers for the important case of linear models. The algorithm presented by Hsieh et al., probably best known under the name of the "liblinear" implementation, marks a major breakthrough. The method is analog to established dual decomposition algorithms for training of non-linear SVMs, but with greatly reduced computational complexity per update step. This comes at the cost of not keeping track of the gradient of the objective any more, which excludes the application of highly developed working set selection algorithms. We present an algorithmic improvement to this method. We replace uniform working set selection with an online adaptation of selection frequencies. The adaptation criterion is inspired by modern second order working set selection methods. The same mechanism replaces the shrinking heuristic. This novel technique speeds up training in some cases by more than an order of magnitude.