XTab: Cross-table Pretraining for Tabular Transformers

TL;DR

XTab introduces a cross-table pretraining framework for tabular transformers that leverages multiple datasets to improve generalization, learning speed, and performance across diverse tabular prediction tasks.

Contribution

The paper presents a novel cross-table pretraining method for tabular transformers, addressing the challenge of inconsistent column types and enabling transfer learning across datasets.

Findings

XTab improves generalization and learning speed of tabular transformers.

Pretraining with XTab outperforms state-of-the-art models on various tasks.

XTab demonstrates consistent performance gains across 84 benchmark datasets.

Abstract

The success of self-supervised learning in computer vision and natural language processing has motivated pretraining methods on tabular data. However, most existing tabular self-supervised learning models fail to leverage information across multiple data tables and cannot generalize to new tables. In this work, we introduce XTab, a framework for cross-table pretraining of tabular transformers on datasets from various domains. We address the challenge of inconsistent column types and quantities among tables by utilizing independent featurizers and using federated learning to pretrain the shared component. Tested on 84 tabular prediction tasks from the OpenML-AutoML Benchmark (AMLB), we show that (1) XTab consistently boosts the generalizability, learning speed, and performance of multiple tabular transformers, (2) by pretraining FT-Transformer via XTab, we achieve superior performance than other state-of-the-art tabular deep learning models on various tasks such as regression, binary, and multiclass classification.