STRABLE: Benchmarking Tabular Machine Learning with Strings

TL;DR

This paper introduces STRABLE, a comprehensive benchmark dataset of real-world tables with strings and numbers, and conducts a large-scale empirical study on various tabular learning pipelines involving string data.

Contribution

It provides the first extensive benchmark for string-including tabular data and evaluates diverse pipelines, revealing insights into effective methods for different types of string data.

Findings

Categorical-dominant tables are well served by simple encodings and advanced learners.

Large LLM encoders are competitive on free-text-dominant tables.

STRABLE enables generalizable pipeline rankings close to oracle rankings.

Abstract

Benchmarking tabular learning has revealed the benefit of dedicated architectures, pushing the state of the art. But real-world tables often contain string entries, beyond numbers, and these settings have been understudied due to a lack of a solid benchmarking suite. They lead to new research questions: Are dedicated learners needed, with end-to-end modeling of strings and numbers? Or does it suffice to encode strings as numbers, as with a categorical encoding? And if so, do the resulting tables resemble numerical tabular data, calling for the same learners? To enable these studies, we contribute STRABLE, a benchmarking corpus of 108 tables, all real-world learning problems with strings and numbers across diverse application fields. We run the first large-scale empirical study of tabular learning with strings, evaluating 445 pipelines. These pipelines span end-to-end architectures and modular pipelines, where strings are first encoded, then post-processed, and finally passed to a tabular learner. We find that, because most tables in the wild are categorical-dominant, advanced tabular learners paired with simple string embeddings achieve good predictions at low computational cost. On free-text-dominant tables, large LLM encoders become competitive. Their performance also appears sensitive to post-processing, with differences across LLM families. Finally, we show that STRABLE is a good set of tables to study "string tabular" learning as it leads to generalizable pipeline rankings that are close to the oracle rankings. We thus establish STRABLE as a foundation for research on tabular learning with strings, an important yet understudied area.