ControlBurn: Nonlinear Feature Selection with Sparse Tree Ensembles

TL;DR

ControlBurn is a Python package that combines nonlinear feature selection with interpretable tree ensemble models, offering scalability, flexibility, and visualization tools for feature analysis in machine learning.

Contribution

It introduces a novel framework for nonlinear feature selection using sparse tree ensembles and a weighted lasso criterion, enhancing interpretability and scalability.

Findings

Supports large datasets with fast regularization path computation

Achieves feature sparsity while maintaining model accuracy

Provides visualization tools for feature impact analysis

Abstract

ControlBurn is a Python package to construct feature-sparse tree ensembles that support nonlinear feature selection and interpretable machine learning. The algorithms in this package first build large tree ensembles that prioritize basis functions with few features and then select a feature-sparse subset of these basis functions using a weighted lasso optimization criterion. The package includes visualizations to analyze the features selected by the ensemble and their impact on predictions. Hence ControlBurn offers the accuracy and flexibility of tree-ensemble models and the interpretability of sparse generalized additive models. ControlBurn is scalable and flexible: for example, it can use warm-start continuation to compute the regularization path (prediction error for any number of selected features) for a dataset with tens of thousands of samples and hundreds of features in seconds. For larger datasets, the runtime scales linearly in the number of samples and features (up to a log factor), and the package support acceleration using sketching. Moreover, the ControlBurn framework accommodates feature costs, feature groupings, and $\ell_0$-based regularizers. The package is user-friendly and open-source: its documentation and source code appear on https://pypi.org/project/ControlBurn/ and https://github.com/udellgroup/controlburn/.