Discrete Tokenization Unlocks Transformers for Calibrated Tabular Forecasting

TL;DR

This paper demonstrates that simple discretized tokenization enables Transformers to excel in tabular forecasting, outperforming gradient boosting methods when combined with Gaussian smoothing for calibration.

Contribution

It introduces a basic tokenization approach that unlocks attention mechanisms for tabular data, surpassing tuned gradient boosting models in accuracy and calibration.

Findings

Transformers outperform gradient boosting on tabular benchmarks.

Discretized tokenization combined with Gaussian smoothing improves calibration.

Ablation studies highlight the importance of architecture choices.

Abstract

Gradient boosting still dominates Transformers on tabular benchmarks. Our tokenizer uses a deliberately simplistic discretized vocabulary so we can highlight how even basic tokenization unlocks the power of attention on tabular features, yet it already outperforms tuned gradient boosting when combined with Gaussian smoothing. Our solution discretizes environmental context while smoothing labels with adaptive Gaussians, yielding calibrated PDFs. On 600K entities (5M training examples) we outperform tuned XGBoost by 10.8% (35.94s vs 40.31s median MAE) and achieve KS=0.0045 with the adaptive-sigma checkpoint selected to minimize KS rather than median MAE. Ablations confirm architecture matters: losing sequential ordering costs about 2.0%, dropping the time-delta tokens costs about 1.8%, and a stratified calibration analysis reveals where miscalibration persists.