Valid inferential models for prediction in supervised learning problems

TL;DR

This paper advocates for the use of valid probabilistic predictors in supervised learning, which provide reliable uncertainty quantification and connect to conformal prediction, enhancing frequentist error control.

Contribution

It introduces a formal notion of validity for probabilistic predictors, demonstrates their imprecision, and offers a general construction linked to conformal prediction for regression and classification.

Findings

Valid probabilistic predictors must be imprecise

They avoid sure loss and ensure error rate control

The construction is applicable to regression and classification

Abstract

Prediction, where observed data is used to quantify uncertainty about a future observation, is a fundamental problem in statistics. Prediction sets with coverage probability guarantees are a common solution, but these do not provide probabilistic uncertainty quantification in the sense of assigning beliefs to relevant assertions about the future observable. Alternatively, we recommend the use of a {\em probabilistic predictor}, a data-dependent (imprecise) probability distribution for the to-be-predicted observation given the observed data. It is essential that the probabilistic predictor be reliable or valid, and here we offer a notion of validity and explore its behavioral and statistical implications. In particular, we show that valid probabilistic predictors must be imprecise, that they avoid sure loss, and that they lead to prediction procedures with desirable frequentist error rate control properties. We provide a general construction of a provably valid probabilistic predictor, which has close connections to the powerful conformal prediction machinery, and we illustrate this construction in regression and classification applications.