A safe Hosmer-Lemeshow test

TL;DR

This paper introduces a new calibration test for binary event predictions using e-values and online isotonic regression, offering improved stability and power over the traditional Hosmer-Lemeshow test, demonstrated through simulations and real data.

Contribution

It proposes an e-value based calibration test that is theoretically superior and more stable than the Hosmer-Lemeshow test, addressing its known limitations.

Findings

The eHL test has power against all alternatives.

It detects slight miscalibrations in practical sample sizes.

It provides clearer results on credit default predictions during a crisis.

Abstract

This article proposes an alternative to the Hosmer-Lemeshow (HL) test for evaluating the calibration of probability forecasts for binary events. The approach is based on e-values, a new tool for hypothesis testing. An e-value is a random variable with expected value less or equal to one under a null hypothesis. Large e-values give evidence against the null hypothesis, and the multiplicative inverse of an e-value is a p-value. Our test uses online isotonic regression to estimate the calibration curve as a `betting strategy' against the null hypothesis. We show that the test has power against essentially all alternatives, which makes it theoretically superior to the HL test and at the same time resolves the well-known instability problem of the latter. A simulation study shows that a feasible version of the proposed eHL test can detect slight miscalibrations in practically relevant sample sizes, but trades its universal validity and power guarantees against a reduced empirical power compared to the HL test in a classical simulation setup.We illustrate our test on recalibrated predictions for credit card defaults during the Taiwan credit card crisis, where the classical HL test delivers equivocal results.