Relational In-Context Learning via Synthetic Pre-training with Structural Prior

TL;DR

This paper introduces RDB-PFN, a synthetic-data trained relational foundation model that excels at in-context learning for real-world relational tasks, overcoming data scarcity issues.

Contribution

The authors develop RDB-PFN, the first relational foundation model trained solely on synthetic data, enabling instant adaptation to new databases through in-context learning.

Findings

RDB-PFN achieves strong few-shot performance on 19 real-world tasks.

It outperforms graph-based and single-table baselines with the same input format.

The model uses a lightweight architecture and offers fast inference.

Abstract

Relational Databases (RDBs) are the backbone of modern business, yet they lack foundation models comparable to those in text or vision. A key obstacle is that high-quality RDBs are private, scarce and structurally heterogeneous, making internet-scale pre-training infeasible. To overcome this data scarcity, We introduce $\textbf{RDB-PFN}$, the first relational foundation model trained purely via $\textbf{synthetic data}$. Inspired by Prior-Data Fitted Networks (PFNs) where synthetic data generated from Structural Causal Models (SCMs) enables reasoning on single tables, we design a $\textbf{Relational Prior Generator}$ to create an infinite stream of diverse RDBs from scratch. Pre-training on $\textbf{over 2 million}$ synthetic single-table and relational tasks, RDB-PFN learns to adapt to any new database instantly via genuine $\textbf{in-context learning}$. Experiments verify RDB-PFN achieves strong few-shot performance on 19 real-world relational prediction tasks, outperforming graph-based and single-table foundation-model baselines (given the same DFS-linearized inputs), while using a lightweight architecture and fast inference. The code is available at https://github.com/MuLabPKU/RDBPFN