Light-Weight Benchmarks Reveal the Hidden Hardware Cost of Zero-Shot Tabular Foundation Models

TL;DR

This paper benchmarks zero-shot tabular foundation models, revealing their significant hardware costs compared to traditional tree models, and provides a reproducible framework for future efficiency research.

Contribution

It introduces a comprehensive benchmark measuring accuracy and hardware metrics for tabular FMs, highlighting their hardware-accuracy trade-offs and establishing a baseline for future work.

Findings

Tree ensembles match or outperform FMs in accuracy.

FMs require substantially more latency and VRAM.

Benchmark provides reproducible hardware and accuracy metrics.

Abstract

Zero-shot foundation models (FMs) promise training-free prediction on tabular data, yet their hardware footprint remains poorly characterized. We present a fully reproducible benchmark that reports test accuracy together with wall-clock latency, peak CPU RAM, and peak GPU VRAM on four public datasets: Adult-Income, Higgs-100k, Wine-Quality, and California-Housing. Two open FMs (TabPFN-1.0 and TabICL-base) are compared against tuned XGBoost, LightGBM, and Random Forest baselines on a single NVIDIA T4 GPU. The tree ensembles equal or surpass FM accuracy on three datasets while completing full-test batches in <= 0.40 s and <= 150 MB RAM, using zero VRAM. TabICL achieves a 0.8 percentage-point gain on Higgs but requires roughly 40,000 times more latency (960 s) and 9 GB VRAM. TabPFN matches tree-model accuracy on Wine and Housing but peaks at 4 GB VRAM and cannot process the full 100k-row Higgs table. These results quantify the substantial hardware-versus-accuracy trade-offs in current tabular FMs and provide an open baseline for future efficiency-oriented research.