Table Foundation Models: on knowledge pre-training for tabular learning

TL;DR

This paper introduces TARTE, a knowledge-enhanced foundation model for tabular data that improves prediction accuracy and efficiency by providing versatile, pre-trained representations capable of fine-tuning and domain specialization.

Contribution

TARTE is a novel pre-trained foundation model for tables that captures semantics through string-based representations, enabling better downstream performance and reusability.

Findings

TARTE improves state-of-the-art prediction accuracy on tabular tasks.

Pre-trained representations facilitate efficient downstream learning.

TARTE enables domain-specific adaptation with minimal additional training.

Abstract

Table foundation models bring high hopes to data science: pre-trained on tabular data to embark knowledge or priors, they should facilitate downstream tasks on tables. One specific challenge is that of data semantics: numerical entries take their meaning from context, e.g., column name. Pre-trained neural networks that jointly model column names and table entries have recently boosted prediction accuracy. While these models outline the promises of world knowledge to interpret table values, they lack the convenience of popular foundation models in text or vision. Indeed, they must be fine-tuned to bring benefits, come with sizeable computation costs, and cannot easily be reused or combined with other architectures. Here we introduce TARTE, a foundation model that transforms tables to knowledge-enhanced vector representations using the string to capture semantics. Pre-trained on large relational data, TARTE yields representations that facilitate subsequent learning with little additional cost. These representations can be fine-tuned or combined with other learners, giving models that push the state-of-the-art prediction performance and improve the prediction/computation performance trade-off. Specialized to a task or a domain, TARTE gives domain-specific representations that facilitate further learning. Our study demonstrates an effective approach to knowledge pre-training for tabular learning.