Obtaining Calibrated Probabilities from Boosting

TL;DR

This paper investigates the calibration issues of boosting algorithms, compares three calibration methods, and explores the impact of different loss functions on probability estimation accuracy.

Contribution

It provides an empirical analysis of probability distortion in AdaBoost and evaluates calibration techniques and loss functions to improve probability estimates.

Findings

Logistic Correction and log-loss boosting perform well with weak models.

Platt Scaling and Isotonic Regression significantly improve probability calibration.

Calibration methods vary in effectiveness depending on model complexity.

Abstract

Boosted decision trees typically yield good accuracy, precision, and ROC area. However, because the outputs from boosting are not well calibrated posterior probabilities, boosting yields poor squared error and cross-entropy. We empirically demonstrate why AdaBoost predicts distorted probabilities and examine three calibration methods for correcting this distortion: Platt Scaling, Isotonic Regression, and Logistic Correction. We also experiment with boosting using log-loss instead of the usual exponential loss. Experiments show that Logistic Correction and boosting with log-loss work well when boosting weak models such as decision stumps, but yield poor performance when boosting more complex models such as full decision trees. Platt Scaling and Isotonic Regression, however, significantly improve the probabilities predicted by