Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifiers

TL;DR

This paper enhances ICL-transformers for tabular data classification by fine-tuning, introducing a new pretraining dataset generator, and combining datasets to improve both fine-tuning and zero-shot performance.

Contribution

It extends TabPFN to fine-tuning, proposes a new dataset generator for complex decision boundaries, and combines datasets for improved overall performance.

Findings

Fine-tuning significantly boosts ICL-transformer performance.

Pretraining on complex datasets improves fine-tuning results.

Combining dataset generators yields state-of-the-art performance.

Abstract

The recently introduced TabPFN pretrains an In-Context Learning (ICL) transformer on synthetic data to perform tabular data classification. In this work, we extend TabPFN to the fine-tuning setting, resulting in a significant performance boost. We also discover that fine-tuning enables ICL-transformers to create complex decision boundaries, a property regular neural networks do not have. Based on this observation, we propose to pretrain ICL-transformers on a new forest dataset generator which creates datasets that are unrealistic, but have complex decision boundaries. TabForest, the ICL-transformer pretrained on this dataset generator, shows better fine-tuning performance when pretrained on more complex datasets. Additionally, TabForest outperforms TabPFN on some real-world datasets when fine-tuning, despite having lower zero-shot performance due to the unrealistic nature of the pretraining datasets. By combining both dataset generators, we create TabForestPFN, an ICL-transformer that achieves excellent fine-tuning performance and good zero-shot performance.