Towards Foundation Models for Relational Databases with Language Models and Graph Neural Networks

TL;DR

This paper introduces a hybrid model combining language models and graph neural networks to improve relational database understanding, showing promising results compared to traditional methods.

Contribution

It proposes a novel hybrid architecture that integrates a fine-tuned BART encoder with a GNN over relational entity graphs, enhancing relational data modeling.

Findings

GNN significantly enriches BART's row embeddings.

Achieves ROC-AUC of 67.40 on rel-f1 dataset.

Performance is competitive with some supervised baselines.

Abstract

Relational databases store much of the world's structured information, and they are essential for driving complex predictive applications. However, deep learning progress on relational data remains limited, as conventional approaches flatten databases into single tables via manual feature engineering, discarding relational context. Relational deep learning (RDL) addresses this by modeling databases as relational entity graphs (REGs) for graph neural networks (GNNs), but remains task- and database-specific. To combine the strengths of both paradigms, we propose a hybrid architecture combining a fine-tuned BART encoder to capture intra-row semantics with a GraphSAGE-based GNN over REGs to inject relational context. Experiments on RelBench show that the GNN substantially enriches BART's row embeddings, achieving a ROC-AUC of 67.40 on the driver-dnf task from the rel-f1 dataset. This performance is competitive with supervised baselines such as LightGBM (68.86) and narrows the gap to RDL (72.62) to within 5.22 points, though a substantial gap remains to state-of-the-art foundation models such as KumoRFM (82.63). These results suggest that lightweight hybrid LM-GNN architectures offer a promising and resource-efficient path towards foundation models for relational databases.