From Tables to Signals: Revealing Spectral Adaptivity in TabPFN

TL;DR

This paper analyzes TabPFN's spectral properties, revealing its broad frequency capacity and spectral adaptivity, which enable training-free tasks like image denoising, advancing understanding of tabular foundation models.

Contribution

It provides the first frequency-based analysis of TabPFN, uncovering its spectral adaptivity and the influence of positional encoding on its frequency response.

Findings

TabPFN has broader effective frequency capacity than standard MLPs.

Spectral capacity of TabPFN adapts to the number of in-context samples.

TabPFN can perform training-free image denoising.

Abstract

Task-agnostic tabular foundation models such as TabPFN have achieved impressive performance on tabular learning tasks, yet the origins of their inductive biases remain poorly understood. In this work, we study TabPFN through the lens of signal reconstruction and provide the first frequency-based analysis of its in-context learning behavior. We show that TabPFN possesses a broader effective frequency capacity than standard ReLU-MLPs, even without hyperparameter tuning. Moreover, unlike MLPs whose spectra evolve primarily over training epochs, we find that TabPFN's spectral capacity adapts directly to the number of samples provided in-context, a phenomenon we term Spectral Adaptivity. We further demonstrate that positional encoding modulates TabPFN's frequency response, mirroring classical results in implicit neural representations. Finally, we show that these properties enable TabPFN to perform training-free and hyperparameter-free image denoising, illustrating its potential as a task-agnostic implicit model. Our analysis provides new insight into the structure and inductive biases of tabular foundation models and highlights their promise for broader signal reconstruction tasks.