Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models

TL;DR

Mitra is a new tabular foundation model trained on a curated mixture of synthetic priors, which significantly improves generalization and sample efficiency over existing models in classification and regression tasks.

Contribution

This work systematically investigates synthetic prior properties and introduces Mitra, the first TFM trained on diverse synthetic priors to enhance real-world data performance.

Findings

Mitra outperforms state-of-the-art TFMs like TabPFNv2 and TabICL.

Mitra demonstrates better sample efficiency in benchmarks.

Synthetic priors with specific properties improve TFM generalization.

Abstract

Since the seminal work of TabPFN, research on tabular foundation models (TFMs) based on in-context learning (ICL) has challenged long-standing paradigms in machine learning. Without seeing any real-world data, models pretrained on purely synthetic datasets generalize remarkably well across diverse datasets, often using only a moderate number of in-context examples. This shifts the focus in tabular machine learning from model architecture design to the design of synthetic datasets, or, more precisely, to the prior distributions that generate them. Yet the guiding principles for prior design remain poorly understood. This work marks the first attempt to address the gap. We systematically investigate and identify key properties of synthetic priors that allow pretrained TFMs to generalize well. Based on these insights, we introduce Mitra, a TFM trained on a curated mixture of synthetic priors selected for their diversity, distinctiveness, and performance on real-world tabular data. Mitra consistently outperforms state-of-the-art TFMs, such as TabPFNv2 and TabICL, across both classification and regression benchmarks, with better sample efficiency.