Towards reliable predictive analytics: a generalized calibration framework

TL;DR

This paper extends calibration methods from binary outcomes to all exponential family distributions, proposing new estimation techniques and performance measures for more reliable predictive analytics.

Contribution

It introduces a generalized calibration framework applicable to various distribution types, along with two estimation methods and new calibration performance metrics.

Findings

The generalized calibration framework effectively assesses different prediction models.

Proposed methods include a generalized linear model and a non-parametric smoother.

Illustrative example demonstrates practical application of the framework.

Abstract

Calibration is a pivotal aspect in predictive modeling, as it ensures that the predictions closely correspond with what we observe empirically. The contemporary calibration framework, however, is predominantly focused on prediction models where the outcome is a binary variable. We extend the logistic calibration framework to the generalized calibration framework which includes all members of the exponential family of distributions. We propose two different methods to estimate the calibration curve in this setting, a generalized linear model and a non-parametric smoother. In addition, we define two measures that summarize the calibration performance. The generalized calibration slope which quantifies the amount of over- or underfitting and the generalized calibration slope or calibration-in-the-large that measures the agreement between the global empirical average and the average predicted value. We provide an illustrative example using a simulated data set and hereby show how we can utilize the generalized calibration framework to assess the calibration of different types of prediction models.