Does TabPFN Understand Causal Structures?

TL;DR

This paper investigates whether the transformer-based TabPFN model, pre-trained on synthetic causal data, encodes causal information in its internal representations, enabling effective causal discovery from tabular data.

Contribution

The study develops an adapter framework to extract causal signals from TabPFN's frozen embeddings, demonstrating its ability to outperform traditional causal discovery methods.

Findings

TabPFN's embeddings contain causal information.

Causal signals are concentrated in mid-range layers.

The approach outperforms several traditional algorithms.

Abstract

Causal discovery is fundamental for multiple scientific domains, yet extracting causal information from real world data remains a significant challenge. Given the recent success on real data, we investigate whether TabPFN, a transformer-based tabular foundation model pre-trained on synthetic datasets generated from structural causal models, encodes causal information in its internal representations. We develop an adapter framework using a learnable decoder and causal tokens that extract causal signals from TabPFN's frozen embeddings and decode them into adjacency matrices for causal discovery. Our evaluations demonstrate that TabPFN's embeddings contain causal information, outperforming several traditional causal discovery algorithms, with such causal information being concentrated in mid-range layers. These findings establish a new direction for interpretable and adaptable foundation models and demonstrate the potential for leveraging pre-trained tabular models for causal discovery.