TreeFlow: Going beyond Tree-based Gaussian Probabilistic Regression

TL;DR

TreeFlow introduces a novel tree-based method that leverages normalizing flows to model complex, multi-modal probability distributions in regression tasks, surpassing traditional Gaussian-based approaches.

Contribution

It combines tree ensembles with normalizing flows to enable flexible probabilistic regression modeling, extending beyond Gaussian assumptions.

Findings

Achieves state-of-the-art results on multi-modal regression benchmarks.

Provides competitive performance on unimodal datasets.

Successfully models complex, multi-modal output distributions.

Abstract

The tree-based ensembles are known for their outstanding performance in classification and regression problems characterized by feature vectors represented by mixed-type variables from various ranges and domains. However, considering regression problems, they are primarily designed to provide deterministic responses or model the uncertainty of the output with Gaussian or parametric distribution. In this work, we introduce TreeFlow, the tree-based approach that combines the benefits of using tree ensembles with the capabilities of modeling flexible probability distributions using normalizing flows. The main idea of the solution is to use a tree-based model as a feature extractor and combine it with a conditional variant of normalizing flow. Consequently, our approach is capable of modeling complex distributions for the regression outputs. We evaluate the proposed method on challenging regression benchmarks with varying volume, feature characteristics, and target dimensionality. We obtain the SOTA results for both probabilistic and deterministic metrics on datasets with multi-modal target distributions and competitive results on unimodal ones compared to tree-based regression baselines.