Uncertainty in Gradient Boosting via Ensembles

TL;DR

This paper explores methods to quantify uncertainty in gradient boosting models, proposing ensemble-based approaches and a virtual ensemble concept to improve anomaly detection and reduce complexity.

Contribution

It introduces a probabilistic ensemble framework for gradient boosting and proposes a virtual ensemble method to enhance uncertainty estimation with reduced complexity.

Findings

Ensembles of gradient boosting models detect anomalies effectively.

Limited improvement in total uncertainty from ensembles.

Virtual ensemble approach reduces complexity significantly.

Abstract

For many practical, high-risk applications, it is essential to quantify uncertainty in a model's predictions to avoid costly mistakes. While predictive uncertainty is widely studied for neural networks, the topic seems to be under-explored for models based on gradient boosting. However, gradient boosting often achieves state-of-the-art results on tabular data. This work examines a probabilistic ensemble-based framework for deriving uncertainty estimates in the predictions of gradient boosting classification and regression models. We conducted experiments on a range of synthetic and real datasets and investigated the applicability of ensemble approaches to gradient boosting models that are themselves ensembles of decision trees. Our analysis shows that ensembles of gradient boosting models successfully detect anomalous inputs while having limited ability to improve the predicted total uncertainty. Importantly, we also propose a concept of a virtual ensemble to get the benefits of an ensemble via only one gradient boosting model, which significantly reduces complexity.