TabPFGen -- Tabular Data Generation with TabPFN

TL;DR

This paper introduces TabPFGen, a novel energy-based generative model for tabular data that leverages a pre-trained transformer, enabling effective data augmentation, balancing, and imputation without additional training.

Contribution

It transforms a discriminative transformer model into a generative one for tabular data, maintaining in-context learning without extra training or tuning.

Findings

Strong results on data augmentation tasks

Effective class balancing and imputation demonstrated

No additional training required for the generative model

Abstract

Advances in deep generative modelling have not translated well to tabular data. We argue that this is caused by a mismatch in structure between popular generative models and discriminative models of tabular data. We thus devise a technique to turn TabPFN -- a highly performant transformer initially designed for in-context discriminative tabular tasks -- into an energy-based generative model, which we dub TabPFGen. This novel framework leverages the pre-trained TabPFN as part of the energy function and does not require any additional training or hyperparameter tuning, thus inheriting TabPFN's in-context learning capability. We can sample from TabPFGen analogously to other energy-based models. We demonstrate strong results on standard generative modelling tasks, including data augmentation, class-balancing, and imputation, unlocking a new frontier of tabular data generation.