UniPredict: Large Language Models are Universal Tabular Classifiers

TL;DR

This paper introduces UniPredict, a large language model-based approach that acts as a universal predictor for tabular data, capable of handling diverse datasets and tasks with superior performance and adaptability.

Contribution

The paper presents a novel generative modeling approach using LLMs for universal tabular data prediction, trained on multiple datasets to outperform specialized models.

Findings

UniPredict outperforms tree-boosting and neural network baselines by 5.4% to 13.4%.

It demonstrates strong few-shot learning capabilities with over 100% performance gains in low-resource settings.

The model effectively adapts to new tabular prediction tasks with minimal data.

Abstract

Tabular data prediction is a fundamental machine learning task for many applications. Existing methods predominantly employ discriminative modeling and operate under the assumption of a fixed target column, necessitating re-training for every new predictive task. Inspired by the generative power of large language models (LLMs), this paper exploits the idea of building universal tabular data predictors based on generative modeling, namely UniPredict. Here, we demonstrate the scalability of an LLM to extensive tabular datasets, enabling it to comprehend diverse tabular inputs and predict target variables following the provided instructions. Specifically, we train a single LLM on an aggregation of 169 tabular datasets with diverse targets and compare its performance against baselines that are trained on each dataset separately. We observe this versatile UniPredict model demonstrates an advantage over other models, ranging from 5.4% to 13.4%, when compared with the best tree-boosting baseline and the best neural network baseline, respectively. We further test UniPredict in few-shot learning settings on another 62 tabular datasets. Our method achieves strong performance in quickly adapting to new tasks. In low-resource few-shot setup, we observed a 100%+ performance advantage compared with XGBoost, and significant margin over all baselines. We envision that UniPredict sheds light on developing a universal tabular data prediction system that learns from data at scale and serves a wide range of prediction tasks.