Speeding Up Budgeted Stochastic Gradient Descent SVM Training with Precomputed Golden Section Search

TL;DR

This paper introduces a fast lookup method for support vector merging in budgeted SVM training, significantly reducing training time without sacrificing accuracy.

Contribution

It replaces iterative merging with a precomputed lookup using Golden Section Search, speeding up training of budgeted SVMs.

Findings

Support vector merging time reduced by up to 65%

Total training time decreased by 44%

No accuracy loss observed

Abstract

Limiting the model size of a kernel support vector machine to a pre-defined budget is a well-established technique that allows to scale SVM learning and prediction to large-scale data. Its core addition to simple stochastic gradient training is budget maintenance through merging of support vectors. This requires solving an inner optimization problem with an iterative method many times per gradient step. In this paper we replace the iterative procedure with a fast lookup. We manage to reduce the merging time by up to 65% and the total training time by 44% without any loss of accuracy.