Prediction Error Reduction Function as a Variable Importance Score

TL;DR

This paper proposes a new, computationally efficient variable importance score for ensemble learning that is simple, handles both regression and classification, and outperforms or matches the random forest importance measure in various examples.

Contribution

The paper introduces a novel variable importance score function that is simpler, more efficient, and versatile compared to existing methods like random forest importance.

Findings

The new score is computationally more efficient than random forest importance.

It handles both regression and classification seamlessly.

It performs favorably in simulated and real data examples.

Abstract

This paper introduces and develops a novel variable importance score function in the context of ensemble learning and demonstrates its appeal both theoretically and empirically. Our proposed score function is simple and more straightforward than its counterpart proposed in the context of random forest, and by avoiding permutations, it is by design computationally more efficient than the random forest variable importance function. Just like the random forest variable importance function, our score handles both regression and classification seamlessly. One of the distinct advantage of our proposed score is the fact that it offers a natural cut off at zero, with all the positive scores indicating importance and significance, while the negative scores are deemed indications of insignificance. An extra advantage of our proposed score lies in the fact it works very well beyond ensemble of trees and can seamlessly be used with any base learners in the random subspace learning context. Our examples, both simulated and real, demonstrate that our proposed score does compete mostly favorably with the random forest score.