Fast Estimation of Partial Dependence Functions using Trees

TL;DR

This paper introduces FastPD, a fast and consistent tree-based estimator for partial dependence functions that improves computational efficiency and handles correlated features better than existing methods like TreeSHAP.

Contribution

We propose FastPD, a novel tree-based estimator for partial dependence functions that is both efficient and consistent, especially with correlated features.

Findings

FastPD estimates PD functions with linear complexity for moderate-depth trees.

FastPD provides consistent estimates unlike TreeSHAP in correlated feature scenarios.

FastPD enables extraction of various PD-based interpretations, including SHAP and interaction effects.

Abstract

Many existing interpretation methods are based on Partial Dependence (PD) functions that, for a pre-trained machine learning model, capture how a subset of the features affects the predictions by averaging over the remaining features. Notable methods include Shapley additive explanations (SHAP) which computes feature contributions based on a game theoretical interpretation and PD plots (i.e., 1-dim PD functions) that capture average marginal main effects. Recent work has connected these approaches using a functional decomposition and argues that SHAP values can be misleading since they merge main and interaction effects into a single local effect. However, a major advantage of SHAP compared to other PD-based interpretations has been the availability of fast estimation techniques, such as \texttt{TreeSHAP}. In this paper, we propose a new tree-based estimator, \texttt{FastPD}, which efficiently estimates arbitrary PD functions. We show that \texttt{FastPD} consistently estimates the desired population quantity -- in contrast to path-dependent \texttt{TreeSHAP} which is inconsistent when features are correlated. For moderately deep trees, \texttt{FastPD} improves the complexity of existing methods from quadratic to linear in the number of observations. By estimating PD functions for arbitrary feature subsets, \texttt{FastPD} can be used to extract PD-based interpretations such as SHAP, PD plots and higher-order interaction effects.