Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Quality

TL;DR

This paper introduces O'Prior, a compositional realism prior for synthetic task distributions, significantly improving tabular foundation model accuracy and robustness by capturing real-world irregularities.

Contribution

The paper presents O'Prior, a novel synthetic prior design framework that enhances tabular model performance by incorporating diverse, realistic, and stress-aware synthetic distributions.

Findings

O'Prior improves downstream accuracy and robustness across benchmarks.

Diversity, realism, and stress modules each independently contribute to performance.

Synthetic prior design is a key determinant of tabular model quality.

Abstract

What determines the quality of a tabular foundation model? Unlike language or vision, tabular foundation models acquire their inductive biases almost entirely from synthetic pretraining distributions, yet the design of these distributions remains poorly understood. Standard synthetic priors are too well-behaved: they omit the irregularities and failure modes that determine deployment robustness. We introduce O'Prior, a compositional realism prior built around four coupled components: a hierarchical SCM meta-generator spanning diverse functional families; a modular realism engine covering heterogeneous marginals, missingness, and target transforms; an explicit stress module injecting confounding and support-query mismatch; and a curriculum-governed, leakage-safe generation protocol. To isolate prior design as the scientific variable, we hold architecture, optimizer, and compute budget fixed and vary only the synthetic task distribution. O'Prior yields consistent and substantial improvements in downstream accuracy and robustness across real tabular benchmarks, with gains concentrated in regimes characterized by distributional irregularities. Ablations confirm that mechanism diversity, realism composition, and shift-aware stress each contribute independently, their effects are not interchangeable. These results establish synthetic prior construction as a first-order and largely overlooked determinant of tabular foundation model quality