Feature Selection for Ridge Regression with Provable Guarantees

TL;DR

This paper presents deterministic and randomized unsupervised feature selection methods for ridge regression, providing theoretical guarantees and demonstrating improved performance over existing techniques through experiments.

Contribution

It introduces spectral sparsification and leverage-score sampling as novel feature selection techniques with provable guarantees for ridge regression.

Findings

Risk bounds show comparable performance to full feature set

Methods outperform existing feature selection techniques

Experimental validation on synthetic and real datasets

Abstract

We introduce single-set spectral sparsification as a deterministic sampling based feature selection technique for regularized least squares classification, which is the classification analogue to ridge regression. The method is unsupervised and gives worst-case guarantees of the generalization power of the classification function after feature selection with respect to the classification function obtained using all features. We also introduce leverage-score sampling as an unsupervised randomized feature selection method for ridge regression. We provide risk bounds for both single-set spectral sparsification and leverage-score sampling on ridge regression in the fixed design setting and show that the risk in the sampled space is comparable to the risk in the full-feature space. We perform experiments on synthetic and real-world datasets, namely a subset of TechTC-300 datasets, to support our theory. Experimental results indicate that the proposed methods perform better than the existing feature selection methods.