The Probability of Improved Prediction: a new concept in statistical inference

TL;DR

This paper introduces the probability of improved prediction (PIP), a new probabilistic measure for model comparison that aims to address criticisms of p-values and enhance inference and prediction integration.

Contribution

It proposes the PIP concept, develops multiple estimators, and explores their relationships with p-values and mean squared error, supported by simulation and real data application.

Findings

PIP provides a probabilistic framework for model comparison.

Estimators of PIP show promising performance in simulations.

PIP can complement p-values to improve inference reliability.

Abstract

In an attempt to provide an answer to the increasing criticism against p-values and to bridge the gap between statistical inference and prediction modelling, we introduce the probability of improved prediction (PIP). In general, the PIP is a probabilistic measure for comparing two competing models. Three versions of the PIP and several estimators are introduced and the relationships between them, p-values and the mean squared error are investigated. The performance of the estimators is assessed in a simulation study. An application shows how the PIP can support p-values to strengthen the conclusions or possibly point at issues with e.g. replicability.