Causal Pre-training Under the Fairness Lens: An Empirical Study of TabPFN

TL;DR

This paper empirically evaluates TabPFN, a foundation model for tabular data pre-trained on causal synthetic datasets, focusing on its predictive accuracy, fairness, and robustness, and finds that fairness improvements are limited.

Contribution

It provides the first comprehensive empirical analysis of TabPFN's fairness and robustness, revealing limitations in fairness despite causal pre-training.

Findings

TabPFN outperforms baselines in predictive accuracy.

Robustness to spurious correlations is observed.

Fairness improvements are moderate and inconsistent.

Abstract

Foundation models for tabular data, such as the Tabular Prior-data Fitted Network (TabPFN), are pre-trained on a massive number of synthetic datasets generated by structural causal models (SCM). They leverage in-context learning to offer high predictive accuracy in real-world tasks. However, the fairness properties of these foundational models, which incorporate ideas from causal reasoning during pre-training, remain underexplored. In this work, we conduct a comprehensive empirical evaluation of TabPFN and its fine-tuned variants, assessing predictive performance, fairness, and robustness across varying dataset sizes and distributional shifts. Our results reveal that while TabPFN achieves stronger predictive accuracy compared to baselines and exhibits robustness to spurious correlations, improvements in fairness are moderate and inconsistent, particularly under missing-not-at-random (MNAR) covariate shifts. These findings suggest that the causal pre-training in TabPFN is helpful but insufficient for algorithmic fairness, highlighting implications for deploying TabPFN (and similar) models in practice and the need for further fairness interventions.