Variational Boosted Soft Trees
Tristan Cinquin, Tammo Rukat, Philipp Schmidt, Martin Wistuba and, Artur Bekasov

TL;DR
This paper introduces a differentiable Bayesian gradient boosting method using soft decision trees, enhancing uncertainty estimation while maintaining strong predictive performance on tabular data.
Contribution
It presents a novel variational inference approach with soft decision trees for Bayesian GBMs, enabling better uncertainty calibration and computational efficiency.
Findings
Higher test likelihoods on 7/10 regression datasets
Improved out-of-distribution detection in 5/10 datasets
Provides useful uncertainty estimates with maintained accuracy
Abstract
Gradient boosting machines (GBMs) based on decision trees consistently demonstrate state-of-the-art results on regression and classification tasks with tabular data, often outperforming deep neural networks. However, these models do not provide well-calibrated predictive uncertainties, which prevents their use for decision making in high-risk applications. The Bayesian treatment is known to improve predictive uncertainty calibration, but previously proposed Bayesian GBM methods are either computationally expensive, or resort to crude approximations. Variational inference is often used to implement Bayesian neural networks, but is difficult to apply to GBMs, because the decision trees used as weak learners are non-differentiable. In this paper, we propose to implement Bayesian GBMs using variational inference with soft decision trees, a fully differentiable alternative to standard…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Machine Learning and Data Classification · Explainable Artificial Intelligence (XAI)
MethodsTest · Variational Inference
