Oblique Bayesian additive regression trees

TL;DR

This paper introduces an oblique Bayesian Additive Regression Trees (BART) model that uses linear combinations of features for decision rules, improving prediction accuracy over traditional axis-aligned BART in various datasets.

Contribution

The paper develops an oblique BART model with a data-adaptive prior, enabling recursive partitioning along hyperplanes, which enhances predictive performance.

Findings

Oblique BART is competitive with existing methods.

Oblique BART often outperforms axis-aligned BART.

The method shows strong results on synthetic and real datasets.

Abstract

Current implementations of Bayesian Additive Regression Trees (BART) are based on axis-aligned decision rules that recursively partition the feature space using a single feature at a time. Several authors have demonstrated that oblique trees, whose decision rules are based on linear combinations of features, can sometimes yield better predictions than axis-aligned trees and exhibit excellent theoretical properties. We develop an oblique version of BART that leverages a data-adaptive decision rule prior that recursively partitions the feature space along random hyperplanes. Using several synthetic and real-world benchmark datasets, we systematically compared our oblique BART implementation to axis-aligned BART and other tree ensemble methods, finding that oblique BART was competitive with -- and sometimes much better than -- those methods.