Inductive Conformal Prediction: A Straightforward Introduction with Examples in Python

TL;DR

This paper introduces Inductive Conformal Prediction (ICP), a distribution-free, model-agnostic method that provides confidence-calibrated prediction sets, illustrated with Python examples for practical understanding.

Contribution

It offers a straightforward, hands-on introduction to ICP, including practical Python examples, emphasizing its importance for high-risk decision-making scenarios.

Findings

ICP guarantees coverage probability without distribution assumptions

Models calibrated with ICP output prediction sets or intervals

ICP enhances interpretability and reliability of predictions in critical applications

Abstract

Inductive Conformal Prediction (ICP) is a set of distribution-free and model agnostic algorithms devised to predict with a user-defined confidence with coverage guarantee. Instead of having point predictions, i.e., a real number in the case of regression or a single class in multi class classification, models calibrated using ICP output an interval or a set of classes, respectively. ICP takes special importance in high-risk settings where we want the true output to belong to the prediction set with high probability. As an example, a classification model might output that given a magnetic resonance image a patient has no latent diseases to report. However, this model output was based on the most likely class, the second most likely class might tell that the patient has a 15% chance of brain tumor or other severe disease and therefore further exams should be conducted. Using ICP is therefore way more informative and we believe that should be the standard way of producing forecasts. This paper is a hands-on introduction, this means that we will provide examples as we introduce the theory.