Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression

TL;DR

This paper systematically evaluates tabular foundation models like TabPFN and TabICL for conditional density estimation, demonstrating their strong performance across diverse datasets and sizes, and highlighting their potential as off-the-shelf CDE tools.

Contribution

It provides the first comprehensive benchmarking of tabular foundation models for CDE, comparing them to traditional methods across multiple metrics and datasets.

Findings

Foundation models outperform most baselines in density accuracy and likelihood.

Calibration is competitive at small sample sizes but varies at larger sizes.

TabPFN outperforms all baselines in a photometric redshift case study.

Abstract

Conditional density estimation (CDE) - recovering the full conditional distribution of a response given tabular covariates - is essential in settings with heteroscedasticity, multimodality, or asymmetric uncertainty. Recent tabular foundation models, such as TabPFN and TabICL, naturally produce predictive distributions, but their effectiveness as general-purpose CDE methods has not been systematically evaluated, unlike their performance for point prediction, which is well studied. We benchmark three tabular foundation model variants against a diverse set of parametric, tree-based, and neural CDE baselines on 39 real-world datasets, across training sizes from 50 to 20,000, using six metrics covering density accuracy, calibration, and computation time. Across all sample sizes, foundation models achieve the best CDE loss, log-likelihood, and CRPS on the large majority of datasets tested. Calibration is competitive at small sample sizes but, for some metrics and datasets, lags behind task-specific neural baselines at larger sample sizes, suggesting that post-hoc recalibration may be a valuable complement. In a photometric redshift case study using SDSS DR18, TabPFN exposed to 50,000 training galaxies outperforms all baselines trained on the full 500,000-galaxy dataset. Taken together, these results establish tabular foundation models as strong off-the-shelf conditional density estimators.