Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic Regression

TL;DR

This paper introduces Probabilistic Gradient Boosting Machines (PGBM), a computationally efficient method for generating probabilistic predictions with a single ensemble, improving uncertainty quantification in large-scale regression tasks.

Contribution

PGBM is a novel approach that approximates leaf weights as random variables, enabling probabilistic predictions from a single model without high computational costs.

Findings

PGBM provides accurate probabilistic estimates without sacrificing point prediction quality.

PGBM offers up to several orders of magnitude speedup over existing methods on large datasets.

PGBM improves probabilistic forecasting by up to 300% in complex tasks like hierarchical time series.

Abstract

Gradient Boosting Machines (GBM) are hugely popular for solving tabular data problems. However, practitioners are not only interested in point predictions, but also in probabilistic predictions in order to quantify the uncertainty of the predictions. Creating such probabilistic predictions is difficult with existing GBM-based solutions: they either require training multiple models or they become too computationally expensive to be useful for large-scale settings. We propose Probabilistic Gradient Boosting Machines (PGBM), a method to create probabilistic predictions with a single ensemble of decision trees in a computationally efficient manner. PGBM approximates the leaf weights in a decision tree as a random variable, and approximates the mean and variance of each sample in a dataset via stochastic tree ensemble update equations. These learned moments allow us to subsequently sample from a specified distribution after training. We empirically demonstrate the advantages of PGBM compared to existing state-of-the-art methods: (i) PGBM enables probabilistic estimates without compromising on point performance in a single model, (ii) PGBM learns probabilistic estimates via a single model only (and without requiring multi-parameter boosting), and thereby offers a speedup of up to several orders of magnitude over existing state-of-the-art methods on large datasets, and (iii) PGBM achieves accurate probabilistic estimates in tasks with complex differentiable loss functions, such as hierarchical time series problems, where we observed up to 10% improvement in point forecasting performance and up to 300% improvement in probabilistic forecasting performance.