Reliable and scalable variable importance estimation via warm-start and early stopping

TL;DR

This paper introduces a scalable method for estimating variable importance in complex models using early stopping and warm-start techniques, providing theoretical guarantees and demonstrating improved efficiency and accuracy.

Contribution

It develops a novel approach combining early stopping and warm-start to efficiently estimate variable importance for gradient-based models, with theoretical guarantees.

Findings

Method reduces computational cost compared to full re-training.

Theoretical guarantees for the proposed variable importance estimation.

Empirical results show improved accuracy and efficiency.

Abstract

As opaque black-box predictive models become more prevalent, the need to develop interpretations for these models is of great interest. The concept of variable importance and Shapley values are interpretability measures that applies to any predictive model and assesses how much a variable or set of variables improves prediction performance. When the number of variables is large, estimating variable importance presents a significant computational challenge because re-training neural networks or other black-box algorithms requires significant additional computation. In this paper, we address this challenge for algorithms using gradient descent and gradient boosting (e.g. neural networks, gradient-boosted decision trees). By using the ideas of early stopping of gradient-based methods in combination with warm-start using the dropout method, we develop a scalable method to estimate variable importance for any algorithm that can be expressed as an iterative kernel update equation. Importantly, we provide theoretical guarantees by using the theory for early stopping of kernel-based methods for neural networks with sufficiently large (but not necessarily infinite) width and gradient-boosting decision trees that use symmetric trees as a weaker learner. We also demonstrate the efficacy of our methods through simulations and a real data example which illustrates the computational benefit of early stopping rather than fully re-training the model as well as the increased accuracy of our approach.