Conditional Density Estimation with Histogram Trees

TL;DR

This paper introduces Conditional Density Trees (CDTrees), a non-parametric, interpretable tree-based model for conditional density estimation that outperforms existing methods in accuracy and robustness.

Contribution

The paper proposes CDTrees, a novel tree-based approach for CDE that uses histogram leaves and MDL for hyperparameter-free learning, enhancing interpretability and performance.

Findings

CDTrees outperform existing interpretable CDE methods in log-loss.

CDTrees are more robust against irrelevant features.

CDTrees produce smaller, more interpretable trees.

Abstract

Conditional density estimation (CDE) goes beyond regression by modeling the full conditional distribution, providing a richer understanding of the data than just the conditional mean in regression. This makes CDE particularly useful in critical application domains. However, interpretable CDE methods are understudied. Current methods typically employ kernel-based approaches, using kernel functions directly for kernel density estimation or as basis functions in linear models. In contrast, despite their conceptual simplicity and visualization suitability, tree-based methods -- which are arguably more comprehensible -- have been largely overlooked for CDE tasks. Thus, we propose the Conditional Density Tree (CDTree), a fully non-parametric model consisting of a decision tree in which each leaf is formed by a histogram model. Specifically, we formalize the problem of learning a CDTree using the minimum description length (MDL) principle, which eliminates the need for tuning the hyperparameter for regularization. Next, we propose an iterative algorithm that, although greedily, searches the optimal histogram for every possible node split. Our experiments demonstrate that, in comparison to existing interpretable CDE methods, CDTrees are both more accurate (as measured by the log-loss) and more robust against irrelevant features. Further, our approach leads to smaller tree sizes than existing tree-based models, which benefits interpretability.