Gradient Boosted Feature Selection

TL;DR

Gradient Boosted Feature Selection (GBFS) is a scalable, flexible algorithm based on Gradient Boosted Trees that effectively identifies relevant features, captures non-linear interactions, and incorporates domain knowledge, outperforming existing methods.

Contribution

This paper introduces GBFS, a novel feature selection method that meets all key criteria including scalability, non-linear interaction detection, and incorporation of prior knowledge, with straightforward implementation.

Findings

GBFS matches or outperforms state-of-the-art algorithms.

It scales efficiently to large datasets.

It allows for domain-specific side information incorporation.

Abstract

A feature selection algorithm should ideally satisfy four conditions: reliably extract relevant features; be able to identify non-linear feature interactions; scale linearly with the number of features and dimensions; allow the incorporation of known sparsity structure. In this work we propose a novel feature selection algorithm, Gradient Boosted Feature Selection (GBFS), which satisfies all four of these requirements. The algorithm is flexible, scalable, and surprisingly straight-forward to implement as it is based on a modification of Gradient Boosted Trees. We evaluate GBFS on several real world data sets and show that it matches or out-performs other state of the art feature selection algorithms. Yet it scales to larger data set sizes and naturally allows for domain-specific side information.