The Backbone Method for Ultra-High Dimensional Sparse Machine Learning

TL;DR

The paper introduces the backbone method, a scalable framework for sparse, interpretable machine learning in ultra-high dimensional settings, significantly reducing computation time while maintaining accuracy.

Contribution

It proposes a two-phase backbone approach that efficiently identifies relevant features, enabling scalable sparse learning and interpretability in extremely high-dimensional data.

Findings

Solves sparse regression with 10^7 features in minutes

Handles decision trees with 10^5 features in minutes

Outperforms or matches state-of-the-art methods in high-dimensional tasks

Abstract

We present the backbone method, a generic framework that enables sparse and interpretable supervised machine learning methods to scale to ultra-high dimensional problems. We solve sparse regression problems with $10^7$ features in minutes and $10^8$ features in hours, as well as decision tree problems with $10^5$ features in minutes.The proposed method operates in two phases: we first determine the backbone set, consisting of potentially relevant features, by solving a number of tractable subproblems; then, we solve a reduced problem, considering only the backbone features. For the sparse regression problem, our theoretical analysis shows that, under certain assumptions and with high probability, the backbone set consists of the truly relevant features. Numerical experiments on both synthetic and real-world datasets demonstrate that our method outperforms or competes with state-of-the-art methods in ultra-high dimensional problems, and competes with optimal solutions in problems where exact methods scale, both in terms of recovering the truly relevant features and in its out-of-sample predictive performance.