TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields

TL;DR

TabSTAR introduces a novel tabular foundation model that leverages target-aware textual representations and transfer learning to significantly improve performance on tabular data with text fields, surpassing traditional methods.

Contribution

The paper presents TabSTAR, a new architecture that incorporates target-aware textual embeddings and unfreezes a pretrained text encoder for enhanced tabular data modeling.

Findings

Achieves state-of-the-art results on classification benchmarks with text features.

Demonstrates effective transfer learning across multiple datasets.

Exhibits scaling laws in pretraining, indicating potential for further improvements.

Abstract

While deep learning has achieved remarkable success across many domains, it has historically underperformed on tabular learning tasks, which remain dominated by gradient boosting decision trees. However, recent advancements are paving the way for Tabular Foundation Models, which can leverage real-world knowledge and generalize across diverse datasets, particularly when the data contains free-text. Although incorporating language model capabilities into tabular tasks has been explored, most existing methods utilize static, target-agnostic textual representations, limiting their effectiveness. We introduce TabSTAR: a Tabular Foundation Model with Semantically Target-Aware Representations. TabSTAR is designed to enable transfer learning on tabular data with textual features, with an architecture free of dataset-specific parameters. It unfreezes a pretrained text encoder and takes as input target tokens, which provide the model with the context needed to learn task-specific embeddings. TabSTAR achieves state-of-the-art performance for both medium- and large-sized datasets across known benchmarks of classification tasks with text features, and its pretraining phase exhibits scaling laws in the number of datasets, offering a pathway for further performance improvements.