PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models

TL;DR

PluRel is a novel framework for generating synthetic multi-table relational databases that enables scaling laws to be observed in Relational Foundation Models, improving their generalization and pretraining performance.

Contribution

We introduce PluRel, a lightweight, flexible framework for synthesizing complex relational databases, facilitating large-scale pretraining and analysis of RFMs.

Findings

RFM pretraining loss follows power-law scaling with synthetic data size.

Scaling synthetic databases enhances model generalization to real data.

Synthetic pretraining produces strong base models for further training.

Abstract

Relational Foundation Models (RFMs) facilitate data-driven decision-making by learning from complex multi-table databases. However, the diverse relational databases needed to train such models are rarely public due to privacy constraints. While there are methods to generate synthetic tabular data of arbitrary size, incorporating schema structure and primary--foreign key connectivity for multi-table generation remains challenging. Here we introduce PluRel, a framework to synthesize multi-tabular relational databases from scratch. In a step-by-step fashion, PluRel models (1) schemas with directed graphs, (2) inter-table primary-foreign key connectivity with bipartite graphs, and, (3) feature distributions in tables via conditional causal mechanisms. The design space across these stages supports the synthesis of a wide range of diverse databases, while being computationally lightweight. Using PluRel, we observe for the first time that (1) RFM pretraining loss exhibits power-law scaling with the number of synthetic databases and total pretraining tokens, (2) scaling the number of synthetic databases improves generalization to real databases, and (3) synthetic pretraining yields strong base models for continued pretraining on real databases. Overall, our framework and results position synthetic data scaling as a promising paradigm for RFMs.