Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformer

TL;DR

This paper introduces an adversarially pre-trained transformer that enables zero-shot tabular prediction, handling diverse datasets and class sizes, and improves performance and efficiency over existing methods.

Contribution

The paper proposes a novel adversarial pre-training approach and a mixture block architecture for zero-shot tabular prediction, addressing class size limitations and enhancing generalization.

Findings

Matches state-of-the-art on small classification tasks

Enhances performance on benchmark datasets in classification and regression

Maintains under one second runtime on average

Abstract

We present an Adversarially Pre-trained Transformer (APT) that is able to perform zero-shot meta-learning on tabular prediction tasks without pre-training on any real-world dataset, extending on the recent development of Prior-Data Fitted Networks (PFNs) and TabPFN. Specifically, APT is pre-trained with adversarial synthetic data agents, who continue to shift their underlying data generating distribution and deliberately challenge the model with different synthetic datasets. In addition, we propose a mixture block architecture that is able to handle classification tasks with arbitrary number of classes, addressing the class size limitation -- a crucial weakness of prior deep tabular zero-shot learners. In experiments, we show that our framework matches state-of-the-art performance on small classification tasks without filtering on dataset characteristics such as number of classes and number of missing values, while maintaining an average runtime under one second. On common benchmark dataset suites in both classification and regression, we show that adversarial pre-training was able to enhance TabPFN's performance. In our analysis, we demonstrate that the adversarial synthetic data agents were able to generate a more diverse collection of data compared to the ordinary random generator in TabPFN. In addition, we demonstrate that our mixture block neural design has improved generalizability and greatly accelerated pre-training.