Regression Trees Know Calculus

TL;DR

This paper explores how regression trees can be used to estimate gradients of differentiable functions, enabling new applications like sensitivity analysis and uncertainty quantification.

Contribution

It introduces a simple, efficient method to estimate local gradients in regression trees, bridging tree models with differentiable algorithms.

Findings

Gradient estimates improve predictive analysis.

Enables uncertainty quantification with trees.

Enhances interpretability of tree models.

Abstract

Regression trees have emerged as a preeminent tool for solving real-world regression problems due to their ability to deal with nonlinearities, interaction effects and sharp discontinuities. In this article, we rather study regression trees applied to well-behaved, differentiable functions, and determine the relationship between node parameters and the local gradient of the function being approximated. We find a simple estimate of the gradient which can be efficiently computed using quantities exposed by popular tree learning libraries. This allows the tools developed in the context of differentiable algorithms, like neural nets and Gaussian processes, to be deployed to tree-based models. To demonstrate this, we study measures of model sensitivity defined in terms of integrals of gradients and demonstrate how to compute them for regression trees using the proposed gradient estimates. Quantitative and qualitative numerical experiments reveal the capability of gradients estimated by regression trees to improve predictive analysis, solve tasks in uncertainty quantification, and provide interpretation of model behavior.