Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap

TL;DR

This paper evaluates ensemble strategies for tabular foundation models, revealing a diversity ceiling and a calibration trade-off, with cascade stacking slightly outperforming single models at high computational cost.

Contribution

It benchmarks six ensemble strategies on 153 datasets, identifying the near-redundancy of models and analyzing the effects of stacking and calibration.

Findings

Cascade stacking improves accuracy by 0.18% over best single model.

Ensemble diversity is limited, with models forming a near-redundant pool.

Meta-learner stacking sharpens class boundaries but worsens calibration.

Abstract

Tabular foundation models (TFMs) now match or beat tuned gradient-boosted trees on a growing fraction of tabular tasks, but no single TFM wins on every dataset. Ensembling is the go to fix here, and it works less well than expected. Six modern TFMs form a near-redundant pool: their mean pairwise Q-statistic is $0.961$, close enough to $1$ that any convex combination is bounded above. We benchmark six ensemble strategies over six TFMs on 153 OpenML classification tasks. The best ensemble, two-level cascade stacking, buys $+0.18\%$ accuracy over the strongest single TFM at $253\times$ the compute. A Friedman and Nemenyi analysis places three ensembles and the best base TFM in a single equivalence group; three other ensembles are significantly \emph{worse} than the best base. Stacking with a logistic-regression meta-learner is the most striking case: competitive accuracy and ROC-AUC, the worst log-loss rank among the ensembles. The meta-learner improves accuracy by sharpening class boundaries, which destroys calibration. We recommend greedy selection as the practical default.