NGBoost: Natural Gradient Boosting for Probabilistic Prediction

TL;DR

NGBoost introduces a natural gradient boosting method for probabilistic prediction, enabling models to output full probability distributions and quantify uncertainty, with broad applicability and improved performance over existing methods.

Contribution

The paper presents NGBoost, a novel gradient boosting algorithm that handles probabilistic regression with multiparameter distributions using natural gradients, enhancing flexibility and scalability.

Findings

NGBoost matches or exceeds existing probabilistic prediction methods.

It offers flexible use with any base learner and distribution family.

The method improves uncertainty quantification in predictions.

Abstract

We present Natural Gradient Boosting (NGBoost), an algorithm for generic probabilistic prediction via gradient boosting. Typical regression models return a point estimate, conditional on covariates, but probabilistic regression models output a full probability distribution over the outcome space, conditional on the covariates. This allows for predictive uncertainty estimation -- crucial in applications like healthcare and weather forecasting. NGBoost generalizes gradient boosting to probabilistic regression by treating the parameters of the conditional distribution as targets for a multiparameter boosting algorithm. Furthermore, we show how the Natural Gradient is required to correct the training dynamics of our multiparameter boosting approach. NGBoost can be used with any base learner, any family of distributions with continuous parameters, and any scoring rule. NGBoost matches or exceeds the performance of existing methods for probabilistic prediction while offering additional benefits in flexibility, scalability, and usability. An open-source implementation is available at github.com/stanfordmlgroup/ngboost.