Variable importance in binary regression trees and forests

TL;DR

This paper develops a theoretical framework for understanding variable importance and pairwise associations in binary regression trees and forests, providing insights into their properties and applications in high-dimensional data analysis.

Contribution

It introduces a novel theoretical characterization of variable importance in binary trees and extends it to ensembles like random forests, addressing a gap in existing theory.

Findings

The theory applies to random forests and helps interpret variable importance.

Provides a basis for screening variables in high-dimensional data.

Enhances understanding of variable associations in tree-based models.

Abstract

We characterize and study variable importance (VIMP) and pairwise variable associations in binary regression trees. A key component involves the node mean squared error for a quantity we refer to as a maximal subtree. The theory naturally extends from single trees to ensembles of trees and applies to methods like random forests. This is useful because while importance values from random forests are used to screen variables, for example they are used to filter high throughput genomic data in Bioinformatics, very little theory exists about their properties.