But How Does It Work in Theory? Linear SVM with Random Features

TL;DR

This paper provides a theoretical analysis showing that a support vector machine with a limited number of random features can achieve faster learning rates than traditional methods under certain conditions, extending previous results to 0-1 loss.

Contribution

It extends the analysis of random features SVMs to the 0-1 loss and demonstrates how optimized feature maps and reweighted feature selection improve performance.

Findings

RFSVM can outperform standard rates with fewer features

Optimized feature maps enhance learning efficiency

Reweighted feature selection improves experimental results

Abstract

We prove that, under low noise assumptions, the support vector machine with $N\ll m$ random features (RFSVM) can achieve the learning rate faster than $O(1/\sqrt{m})$ on a training set with $m$ samples when an optimized feature map is used. Our work extends the previous fast rate analysis of random features method from least square loss to 0-1 loss. We also show that the reweighted feature selection method, which approximates the optimized feature map, helps improve the performance of RFSVM in experiments on a synthetic data set.