Benchmarking Distribution Shift in Tabular Data with TableShift

TL;DR

TableShift is a new benchmark for evaluating the robustness of tabular data models against distribution shifts across diverse real-world domains, enabling systematic assessment and comparison of model performance under such shifts.

Contribution

The paper introduces TableShift, the first comprehensive benchmark for distribution shift in tabular data, including diverse tasks, data sources, and shifts, along with a large-scale evaluation of models.

Findings

Domain robustness methods can reduce shift gaps but lower in-distribution accuracy.

A linear relationship exists between in-distribution and out-of-distribution accuracy.

Shift gaps correlate with changes in label distribution.

Abstract

Robustness to distribution shift has become a growing concern for text and image models as they transition from research subjects to deployment in the real world. However, high-quality benchmarks for distribution shift in tabular machine learning tasks are still lacking despite the widespread real-world use of tabular data and differences in the models used for tabular data in comparison to text and images. As a consequence, the robustness of tabular models to distribution shift is poorly understood. To address this issue, we introduce TableShift, a distribution shift benchmark for tabular data. TableShift contains 15 binary classification tasks in total, each with an associated shift, and includes a diverse set of data sources, prediction targets, and distribution shifts. The benchmark covers domains including finance, education, public policy, healthcare, and civic participation, and is accessible using only a few lines of Python code via the TableShift API. We conduct a large-scale study comparing several state-of-the-art tabular data models alongside robust learning and domain generalization methods on the benchmark tasks. Our study demonstrates (1) a linear trend between in-distribution (ID) and out-of-distribution (OOD) accuracy; (2) domain robustness methods can reduce shift gaps but at the cost of reduced ID accuracy; (3) a strong relationship between shift gap (difference between ID and OOD performance) and shifts in the label distribution. The benchmark data, Python package, model implementations, and more information about TableShift are available at https://github.com/mlfoundations/tableshift and https://tableshift.org .