Distributional Adaptive Soft Regression Trees

TL;DR

This paper introduces a novel distributional regression tree using multivariate soft split rules, enabling efficient modeling of complex distributions with high-dimensional data, outperforming benchmarks especially in non-linear scenarios.

Contribution

It proposes a new distributional regression tree with soft split rules that simplifies split selection and improves modeling of complex, high-dimensional distributions.

Findings

Outperforms benchmark methods in simulations

Effectively models complex non-linear interactions

Provides probabilistic forecasts for solar activity

Abstract

Random forests are an ensemble method relevant for many problems, such as regression or classification. They are popular due to their good predictive performance (compared to, e.g., decision trees) requiring only minimal tuning of hyperparameters. They are built via aggregation of multiple regression trees during training and are usually calculated recursively using hard splitting rules. Recently regression forests have been incorporated into the framework of distributional regression, a nowadays popular regression approach aiming at estimating complete conditional distributions rather than relating the mean of an output variable to input features only - as done classically. This article proposes a new type of a distributional regression tree using a multivariate soft split rule. One great advantage of the soft split is that smooth high-dimensional functions can be estimated with only one tree while the complexity of the function is controlled adaptive by information criteria. Moreover, the search for the optimal split variable is obsolete. We show by means of extensive simulation studies that the algorithm has excellent properties and outperforms various benchmark methods, especially in the presence of complex non-linear feature interactions. Finally, we illustrate the usefulness of our approach with an example on probabilistic forecasts for the Sun's activity.