Small Language Models for Tabular Data

TL;DR

This paper demonstrates that small language models can effectively perform classification and regression on small, poorly curated tabular datasets by encoding inputs as character sequences, achieving high accuracy and interpretability.

Contribution

It introduces a novel approach of applying small language models to tabular data without explicit feature engineering, showing their capacity for representation learning and input attribution.

Findings

Small models achieve record classification accuracy on tabular data.

Models form useful feature embeddings even without explicit feature knowledge.

Input attribution enables understanding feature importance.

Abstract

Supervised deep learning is most commonly applied to difficult problems defined on large and often extensively curated datasets. Here we demonstrate the ability of deep representation learning to address problems of classification and regression from small and poorly formed tabular datasets by encoding input information as abstracted sequences composed of a fixed number of characters per input field. We find that small models have sufficient capacity for approximation of various functions and achieve record classification benchmark accuracy. Such models are shown to form useful embeddings of various input features in their hidden layers, even if the learned task does not explicitly require knowledge of those features. These models are also amenable to input attribution, allowing for an estimation of the importance of each input element to the model output as well as of which inputs features are effectively embedded in the model. We present a proof-of-concept for the application of small language models to mixed tabular data without explicit feature engineering, cleaning, or preprocessing, relying on the model to perform these tasks as part of the representation learning process.