Cross-Table Pretraining towards a Universal Function Space for Heterogeneous Tabular Data

TL;DR

This paper introduces XTFormer, a cross-table pretrained Transformer that learns a universal function space for diverse tabular data, significantly improving performance across numerous downstream prediction tasks.

Contribution

The paper proposes XTFormer, a novel cross-table pretraining method that creates a meta-function space for versatile tabular prediction, addressing diversity and data scarcity challenges.

Findings

XTFormer outperforms XGBoost and Catboost on 72% of tasks.

It surpasses FT-Transformer and XTab on over 75% of tasks.

Achieves state-of-the-art results across 190 downstream tasks.

Abstract

Tabular data from different tables exhibit significant diversity due to varied definitions and types of features, as well as complex inter-feature and feature-target relationships. Cross-dataset pretraining, which learns reusable patterns from upstream data to support downstream tasks, have shown notable success in various fields. Yet, when applied to tabular data prediction, this paradigm faces challenges due to the limited reusable patterns among diverse tabular datasets (tables) and the general scarcity of tabular data available for fine-tuning. In this study, we fill this gap by introducing a cross-table pretrained Transformer, XTFormer, for versatile downstream tabular prediction tasks. Our methodology insight is pretraining XTFormer to establish a "meta-function" space that encompasses all potential feature-target mappings. In pre-training, a variety of potential mappings are extracted from pre-training tabular datasets and are embedded into the "meta-function" space, and suited mappings are extracted from the "meta-function" space for downstream tasks by a specified coordinate positioning approach. Experiments show that, in 190 downstream tabular prediction tasks, our cross-table pretrained XTFormer wins both XGBoost and Catboost on 137 (72%) tasks, and surpasses representative deep learning models FT-Transformer and the tabular pre-training approach XTab on 144 (76%) and 162 (85%) tasks.