Classifier Calibration at Scale: An Empirical Study of Model-Agnostic Post-Hoc Methods

TL;DR

This study empirically evaluates various post-hoc calibration methods for binary classifiers on tabular data, highlighting the strengths of Venn-Abers and Beta calibration in improving probabilistic predictions while noting the limitations of traditional methods like Platt scaling.

Contribution

It provides a comprehensive benchmark of 21 classifiers and 5 calibration methods, revealing the relative effectiveness and limitations of each in real-world tabular classification tasks.

Findings

Venn-Abers predictors achieve the largest average log-loss reduction.

Beta calibration most frequently improves log-loss across tasks.

Platt scaling and isotonic regression can degrade calibration performance.

Abstract

We study model-agnostic post-hoc calibration methods intended to improve probabilistic predictions in supervised binary classification on real i.i.d. tabular data, with particular emphasis on conformal and Venn-based approaches that provide distribution-free validity guarantees under exchangeability. We benchmark 21 widely used classifiers, including linear models, SVMs, tree ensembles (CatBoost, XGBoost, LightGBM), and modern tabular neural and foundation models, on binary tasks from the TabArena-v0.1 suite using randomized, stratified five-fold cross-validation with a held-out test fold. Five calibrators; Isotonic regression, Platt scaling, Beta calibration, Venn-Abers predictors, and Pearsonify are trained on a separate calibration split and applied to test predictions. Calibration is evaluated using proper scoring rules (log-loss and Brier score) and diagnostic measures (Spiegelhalter's Z, ECE, and ECI), alongside discrimination (AUC-ROC) and standard classification metrics. Across tasks and architectures, Venn-Abers predictors achieve the largest average reductions in log-loss, followed closely by Beta calibration, while Platt scaling exhibits weaker and less consistent effects. Beta calibration improves log-loss most frequently across tasks, whereas Venn-Abers displays fewer instances of extreme degradation and slightly more instances of extreme improvement. Importantly, we find that commonly used calibration procedures, most notably Platt scaling and isotonic regression, can systematically degrade proper scoring performance for strong modern tabular models. Overall classification performance is often preserved, but calibration effects vary substantially across datasets and architectures, and no method dominates uniformly. In expectation, all methods except Pearsonify slightly increase accuracy, but the effect is marginal, with the largest expected gain about 0.008%.