SBAMDT: Bayesian Additive Decision Trees with Adaptive Soft Semi-multivariate Split Rules

TL;DR

This paper introduces SBAMDT, a Bayesian additive decision tree model with adaptive soft split rules that improve flexibility and capture complex decision boundaries by leveraging multivariate features and dynamic smoothness adaptation.

Contribution

The paper presents a novel probabilistic soft split rule for Bayesian additive decision trees, enhancing their ability to model complex, multivariate decision boundaries with adaptive smoothness.

Findings

Outperforms traditional decision trees on synthetic datasets.

Effectively captures complex decision boundaries in spatial data.

Demonstrates improved predictive performance on NYC education data.

Abstract

Bayesian Additive Regression Trees [BART, Chipman et al., 2010] have gained significant popularity due to their remarkable predictive performance and ability to quantify uncertainty. However, standard decision tree models rely on recursive data splits at each decision node, using deterministic decision rules based on a single univariate feature. This approach limits their ability to effectively capture complex decision boundaries, particularly in scenarios involving multiple features, such as spatial domains, or when transitions are either sharp or smoothly varying. In this paper, we introduce a novel probabilistic additive decision tree model that employs a soft split rule. This method enables highly flexible splits that leverage both univariate and multivariate features, while also respecting the geometric properties of the feature domain. Notably, the probabilistic split rule adapts dynamically across decision nodes, allowing the model to account for varying levels of smoothness in the regression function. We demonstrate the utility of the proposed model through comparisons with existing tree-based models on synthetic datasets and a New York City education dataset.