Scaling SVM and Least Absolute Deviations via Exact Data Reduction

TL;DR

This paper introduces a safe, efficient data reduction technique called DVI for SVM and LAD that significantly decreases computational costs by discarding non-support vectors, enabling faster large-scale classification and regression.

Contribution

The paper presents the first screening method for LAD and extends DVI to efficiently identify non-support vectors in SVM and LAD problems.

Findings

DVI can discard up to 99% of data points as non-support vectors.

The method achieves up to 100x speedup in SVM and LAD computations.

DVI outperforms existing screening rules in accuracy and efficiency.

Abstract

The support vector machine (SVM) is a widely used method for classification. Although many efforts have been devoted to develop efficient solvers, it remains challenging to apply SVM to large-scale problems. A nice property of SVM is that the non-support vectors have no effect on the resulting classifier. Motivated by this observation, we present fast and efficient screening rules to discard non-support vectors by analyzing the dual problem of SVM via variational inequalities (DVI). As a result, the number of data instances to be entered into the optimization can be substantially reduced. Some appealing features of our screening method are: (1) DVI is safe in the sense that the vectors discarded by DVI are guaranteed to be non-support vectors; (2) the data set needs to be scanned only once to run the screening, whose computational cost is negligible compared to that of solving the SVM problem; (3) DVI is independent of the solvers and can be integrated with any existing efficient solvers. We also show that the DVI technique can be extended to detect non-support vectors in the least absolute deviations regression (LAD). To the best of our knowledge, there are currently no screening methods for LAD. We have evaluated DVI on both synthetic and real data sets. Experiments indicate that DVI significantly outperforms the existing state-of-the-art screening rules for SVM, and is very effective in discarding non-support vectors for LAD. The speedup gained by DVI rules can be up to two orders of magnitude.