A principled framework for uncertainty decomposition in TabPFN

TL;DR

This paper introduces a Bayesian uncertainty decomposition framework for TabPFN, a transformer model for tabular data, providing fast credible bands and entropy-based uncertainty analysis.

Contribution

It develops a novel asymptotic method for uncertainty decomposition in TabPFN, including variance estimators and credible bands, filling a theoretical gap.

Findings

Credible bands achieve near-nominal coverage.

Variance estimators effectively capture predictive volatility.

Entropy-based decomposition enhances classification uncertainty analysis.

Abstract

TabPFN is a transformer that achieves state-of-the-art performance on supervised tabular tasks by amortizing Bayesian prediction into a single forward pass. However, there is currently no method for uncertainty decomposition in TabPFN. Because it behaves, in an idealised limit, as a Bayesian in-context learner, we cast the decomposition challenge as a Bayesian predictive inference (BPI) problem. The main computational tool in BPI, predictive Monte Carlo, is challenging to apply here as it requires simulating unmodeled covariates. We therefore pursue the asymptotic alternative, filling a gap in the theory for supervised settings by proving a predictive CLT under quasi-martingale conditions. We derive variance estimators determined by the volatility of predictive updates along the context. The resulting credible bands are fast to compute, target epistemic uncertainty, and achieve near-nominal frequentist coverage. For classification, we further obtain an entropy-based uncertainty decomposition.