Multi-Merge Budget Maintenance for Stochastic Gradient Descent SVM Training

TL;DR

This paper introduces a more efficient multi-merge strategy for Budgeted Stochastic Gradient Descent in SVM training, significantly reducing merge computation time while maintaining accuracy.

Contribution

It proposes merging more than two points at once in BSGD, improving training efficiency without loss of model performance.

Findings

Significant speed-ups in training time

Maintained accuracy with multi-merge schemes

Reduced merge operation costs by up to 45%

Abstract

Budgeted Stochastic Gradient Descent (BSGD) is a state-of-the-art technique for training large-scale kernelized support vector machines. The budget constraint is maintained incrementally by merging two points whenever the pre-defined budget is exceeded. The process of finding suitable merge partners is costly; it can account for up to 45% of the total training time. In this paper we investigate computationally more efficient schemes that merge more than two points at once. We obtain significant speed-ups without sacrificing accuracy.