TabularMath: Evaluating Computational Extrapolation in Tabular Learning via Program-Verified Synthesis

TL;DR

This paper introduces TabularMath, a benchmark for evaluating tabular models' ability to perform computational extrapolation on deterministic problems, revealing strengths and limitations of current models like TabPFN and GPT-based ICL.

Contribution

The paper presents a new benchmark, TabularMath, for assessing computational extrapolation in tabular learning, and provides a comparative analysis of nine architectures and GPT-ICL on this benchmark.

Findings

TabPFN achieves high R^2 but low exact match accuracy out-of-distribution.

GPT-OSS-120B ICL maintains better exact match accuracy under distribution shift.

Models excel at smooth function approximation but struggle with precise extrapolation.

Abstract

Standard tabular benchmarks mainly focus on the evaluation of a model's capability to interpolate values inside a data manifold, where models good at performing local statistical smoothing are rewarded. However, there exists a very large category of high-value tabular data, including financial modeling and physical simulations, which are generated based upon deterministic computational processes, as opposed to stochastic and noisy relationships. Therefore, we investigate if tabular models can provide an extension from statistical interpolation to computational extrapolation. We propose TabularMath, a diagnostic benchmark of 114 deterministic problems (233,472 rows) generated from verified programs based on GSM8K and AIME. We evaluate 9 tabular architectures and in-context learning (ICL) with GPT-OSS-120B. On standard regression metrics, TabPFN v2.5 performs remarkably well, achieving R^2=0.998 in-distribution and maintaining positive R^2 even under distribution shift, which is unique among the tabular models we tested. When we measure rounded consistency (exact integer match), a different picture emerges: TabPFN v2.5 drops below 10% on out-of-distribution data, while ICL maintains around 40%. This gap between R^2 and exact-match accuracy suggests that tabular models learn smooth function approximations but struggle to recover precise computational outputs under extrapolation. The two paradigms appear complementary: TabPFN scales efficiently with data; ICL achieves exact computation from few examples. We release all code and data to support further investigation.