KumoRFM-2: Scaling Foundation Models for Relational Learning

TL;DR

KumoRFM-2 is a scalable, pre-trained foundation model for relational data that excels in in-context learning and fine-tuning, outperforming supervised methods on benchmarks and handling billion-scale datasets.

Contribution

Introduces KumoRFM-2, a novel relational foundation model that processes connected tables without flattening, leveraging synthetic and real data for superior performance.

Findings

Outperforms supervised approaches by up to 8% on benchmarks.

Surpasses previous models in cold start and noisy data scenarios.

Scales to billion-scale relational datasets.

Abstract

We introduce KumoRFM-2, the next iteration of a pre-trained foundation model for relational data. KumoRFM-2 supports in-context learning as well as fine-tuning and is applicable to a wide range of predictive tasks. In contrast to tabular foundation models, KumoRFM-2 natively operates on relational data, processing one or more connected tables simultaneously without manual table flattening or target variable generation, all while preserving temporal consistency. KumoRFM-2 leverages a large corpus of synthetic and real-world data to pre-train across four axes: the row and column dimensions at the individual table level, and the foreign key and cross-sample dimensions at the database level. In contrast to its predecessor, KumoRFM-2 injects task information as early as possible, enabling sharper selection of task-relevant columns and improved robustness to noisy data. Through extensive experiments on 41 challenging benchmarks and analysis around expressivity and sensitivity, we demonstrate that KumoRFM-2 outperforms supervised and foundational approaches by up to 8%, while maintaining strong performance under extreme settings of cold start and noisy data. To our knowledge, this is the first time a few-shot foundation model has been shown to surpass supervised approaches on common benchmark tasks, with performance further improving upon fine-tuning. Finally, while KumoRFM-1 was limited to small-scale in-memory datasets, KumoRFM-2 scales to billion-scale relational datasets.