Basis Transformers for Multi-Task Tabular Regression

TL;DR

This paper introduces basis transformers, a new architecture tailored for multi-task tabular regression that effectively handles data heterogeneity, noise, and unseen features, outperforming existing models in accuracy and efficiency.

Contribution

The paper presents basis transformers, a novel model architecture designed specifically for tabular data, addressing challenges like heterogeneity, partial information, and invariances, with superior performance and fewer parameters.

Findings

Improved median R^2 score by 0.338 on OpenML-CTR23 benchmark.

Achieved lowest standard deviation across 34 tasks.

Model has five times fewer parameters than top baselines.

Abstract

Dealing with tabular data is challenging due to partial information, noise, and heterogeneous structure. Existing techniques often struggle to simultaneously address key aspects of tabular data such as textual information, a variable number of columns, and unseen data without metadata besides column names. We propose a novel architecture, \textit{basis transformers}, specifically designed to tackle these challenges while respecting inherent invariances in tabular data, including hierarchical structure and the representation of numeric values. We evaluate our design on a multi-task tabular regression benchmark, achieving an improvement of 0.338 in the median $R^2$ score and the lowest standard deviation across 34 tasks from the OpenML-CTR23 benchmark. Furthermore, our model has five times fewer parameters than the best-performing baseline and surpasses pretrained large language model baselines -- even when initialized from randomized weights.