Transfer Learning of Tabular Data by Finetuning Large Language Models

TL;DR

This paper explores finetuning large language models for tabular data classification, demonstrating superior performance and efficiency on small-feature datasets through transfer learning.

Contribution

It introduces an end-to-end finetuning method for LLMs tailored to tabular data, enabling effective transfer learning without existing large pre-trained models.

Findings

Outperforms state-of-the-art methods on small-feature datasets

Uses less computational cost than other deep learning approaches

Achieves competitive or superior classification accuracy

Abstract

Despite the artificial intelligence (AI) revolution, deep learning has yet to achieve much success with tabular data due to heterogeneous feature space and limited sample sizes without viable transfer learning. The new era of generative AI, powered by large language models (LLM), brings unprecedented learning opportunities to diverse data and domains. This paper investigates the effectiveness of an LLM application programming interface (API) and transfer learning of LLM in tabular data classification. LLM APIs respond to input text prompts with tokenized data and instructions, whereas transfer learning finetunes an LLM for a target classification task. This paper proposes an end-to-end finetuning of LLM to demonstrate cross-data transfer learning on ten benchmark data sets when large pre-trained tabular data models do not exist to facilitate transfer learning. The proposed LLM finetuning method outperforms state-of-the-art machine and deep learning methods on tabular data with less than ten features - a standard feature size for tabular data sets. The transfer learning approach uses a fraction of the computational cost of other deep learning or API-based solutions while ensuring competitive or superior classification performance.