Multivariate Probabilistic Regression with Natural Gradient Boosting

TL;DR

This paper introduces NGBoost, a natural gradient boosting method for multivariate probabilistic regression that models joint uncertainty, demonstrating robustness and competitive performance in simulations and a real-world oceanographic case study.

Contribution

It presents a novel NGBoost approach for multivariate probabilistic regression that is flexible, robust, and easy to use without extensive tuning.

Findings

Performs competitively against existing methods

Works out-of-the-box with minimal tuning

Effectively models joint uncertainty in multivariate data

Abstract

Many single-target regression problems require estimates of uncertainty along with the point predictions. Probabilistic regression algorithms are well-suited for these tasks. However, the options are much more limited when the prediction target is multivariate and a joint measure of uncertainty is required. For example, in predicting a 2D velocity vector a joint uncertainty would quantify the probability of any vector in the plane, which would be more expressive than two separate uncertainties on the x- and y- components. To enable joint probabilistic regression, we propose a Natural Gradient Boosting (NGBoost) approach based on nonparametrically modeling the conditional parameters of the multivariate predictive distribution. Our method is robust, works out-of-the-box without extensive tuning, is modular with respect to the assumed target distribution, and performs competitively in comparison to existing approaches. We demonstrate these claims in simulation and with a case study predicting two-dimensional oceanographic velocity data. An implementation of our method is available at https://github.com/stanfordmlgroup/ngboost.