MultiTab: A Comprehensive Benchmark Suite for Multi-Dimensional Evaluation in Tabular Domains

TL;DR

MultiTab introduces a benchmark suite that categorizes datasets by data characteristics to evaluate tabular models across diverse regimes, revealing how model performance varies with data properties.

Contribution

It provides a comprehensive, data-aware evaluation framework for tabular learning algorithms, highlighting the importance of regime-specific analysis for model selection and design.

Findings

Model performance varies significantly across data regimes.

Sample-level similarity models excel with large samples or high feature correlation.

Inter-feature dependency models perform best with weakly correlated features.

Abstract

Despite the widespread use of tabular data in real-world applications, most benchmarks rely on average-case metrics, which fail to reveal how model behavior varies across diverse data regimes. To address this, we propose MultiTab, a benchmark suite and evaluation framework for multi-dimensional, data-aware analysis of tabular learning algorithms. Rather than comparing models only in aggregate, MultiTab categorizes 196 publicly available datasets along key data characteristics, including sample size, label imbalance, and feature interaction, and evaluates 13 representative models spanning a range of inductive biases. Our analysis shows that model performance is highly sensitive to such regimes: for example, models using sample-level similarity excel on datasets with large sample sizes or high inter-feature correlation, while models encoding inter-feature dependencies perform best with weakly correlated features. These findings reveal that inductive biases do not always behave as intended, and that regime-aware evaluation is essential for understanding and improving model behavior. MultiTab enables more principled model design and offers practical guidance for selecting models tailored to specific data characteristics. All datasets, code, and optimization logs are publicly available at https://huggingface.co/datasets/LGAI-DILab/Multitab.