Pre-trained Tabular Foundation Models as Versatile Summary Networks for Neural Posterior Estimation

TL;DR

This paper explores using pre-trained tabular foundation models like TabPFN as fixed summary networks for simulation-based Bayesian inference, demonstrating their effectiveness and limitations across various settings.

Contribution

It introduces PFN-NPE, a method that leverages pretrained TabPFN models as summary encoders for SBI, achieving competitive or superior posterior approximations.

Findings

TabPFN-derived summaries often preserve useful posterior information.

PFN-NPE matches or outperforms established methods in posterior approximation.

Summaries may struggle to represent joint posterior structure despite good marginal recovery.

Abstract

In this work, we study TabPFN as a training-free, modular summary network for simulation-based Bayesian inference (SBI). Tabular foundation models such as TabPFN are pretrained on broad families of synthetic tabular data-generating processes and adapt at test time through in-context learning, making them natural candidates for SBI, where posterior estimation often depends on learning informative summaries of simulated observations. We propose PFN-NPE: a general recipe that uses a pretrained TabPFN encoder as a fixed summary network for simulator outputs, then pairs the resulting summaries with a downstream inference head chosen for the problem. With normalizing flows as the default inference head, PFN-NPE matches established posterior approximation methods and sometimes outperforms them. More importantly, diagnostic probes show that the TabPFN-derived summaries often preserve useful posterior location and marginal information. These analyses also reveal a limitation in that TabPFN-derived summaries may struggle to represent the joint posterior structure even when the marginals are well recovered. Still, our experiments show that TabPFN can serve as an effective summary network across a diverse set of SBI settings, with the inference network left modular and task-dependent.