Random Projections for Linear Support Vector Machines

TL;DR

This paper introduces a new oblivious dimension reduction method using random projections for linear SVMs, preserving margins and generalization capabilities with high probability, supported by theoretical proofs and extensive experiments.

Contribution

The paper proposes a novel precomputed random projection technique that preserves SVM margins and geometric properties, applicable to any input data matrix.

Findings

Preserves margin and ball radius within psilon error with high probability

Ensures comparable generalization in classification tasks

Supported by extensive experiments on real and synthetic data

Abstract

Let X be a data matrix of rank \rho, whose rows represent n points in d-dimensional space. The linear support vector machine constructs a hyperplane separator that maximizes the 1-norm soft margin. We develop a new oblivious dimension reduction technique which is precomputed and can be applied to any input matrix X. We prove that, with high probability, the margin and minimum enclosing ball in the feature space are preserved to within \epsilon-relative error, ensuring comparable generalization as in the original space in the case of classification. For regression, we show that the margin is preserved to \epsilon-relative error with high probability. We present extensive experiments with real and synthetic data to support our theory.