On the Gaussian process limit of Bayesian Additive Regression Trees

TL;DR

This paper derives the Gaussian process limit of Bayesian Additive Regression Trees (BART), providing an analytical covariance function and demonstrating how it can be used as a GP surrogate, simplifying inference and enhancing understanding.

Contribution

It introduces the exact BART prior covariance function and implements the infinite trees limit as a Gaussian process, enabling analytical likelihood and easier model building.

Findings

The GP limit of BART is initially worse than standard BART.

Tuning hyperparameters in the GP framework makes it competitive with BART.

The GP surrogate simplifies the modeling process and sidesteps complex MCMC algorithms.

Abstract

Bayesian Additive Regression Trees (BART) is a nonparametric Bayesian regression technique of rising fame. It is a sum-of-decision-trees model, and is in some sense the Bayesian version of boosting. In the limit of infinite trees, it becomes equivalent to Gaussian process (GP) regression. This limit is known but has not yet led to any useful analysis or application. For the first time, I derive and compute the exact BART prior covariance function. With it I implement the infinite trees limit of BART as GP regression. Through empirical tests, I show that this limit is worse than standard BART in a fixed configuration, but also that tuning its hyperparameters in the natural GP way makes it competitive with BART. The advantage of using a GP surrogate of BART is the analytical likelihood, which simplifies model building and sidesteps the complex BART MCMC algorithm. More generally, this study opens new ways to understand and develop BART and GP regression. The implementation of BART as GP is available in the Python package lsqfitgp.