A distribution-free valid p-value for finite samples of bounded random variables

TL;DR

This paper introduces a new distribution-free p-value for finite samples of bounded variables, improving calibration in machine learning and classical inference by leveraging a concentration inequality, and demonstrating its tighter bounds compared to existing methods.

Contribution

It presents a novel super-uniform p-value based on a concentration inequality, offering tighter bounds than Hoeffding and Bentkus in certain regions, applicable to both machine learning and classical statistics.

Findings

The p-value is valid for finite samples of bounded variables.

It is tighter than Hoeffding and Bentkus bounds in specific regions.

The method enhances calibration and inference in distribution-free settings.

Abstract

We build a valid p-value based on a concentration inequality for bounded random variables introduced by Pelekis, Ramon and Wang. The motivation behind this work is the calibration of predictive algorithms in a distribution-free setting. The super-uniform p-value is tighter than Hoeffding and Bentkus alternatives in certain regions. Even though we are motivated by a calibration setting in a machine learning context, the ideas presented in this work are also relevant in classical statistical inference. Furthermore, we compare the power of a collection of valid p- values for bounded losses, which are presented in previous literature.