A classical measure of evidence for general null hypotheses

TL;DR

This paper introduces an objective measure of evidence for general null hypotheses that overcomes limitations of p-values, providing a logical and belief-based framework for significance testing.

Contribution

It proposes a new evidence measure that satisfies logical criteria and functions as a possibility measure, addressing p-value criticisms and enhancing hypothesis assessment.

Findings

The measure is compatible with belief calculus formalism.

It can establish objective states of belief over parameter subsets.

Recommended as an additional summary for significance tests.

Abstract

In science, the most widespread statistical quantities are perhaps $p$-values. A typical advice is to reject the null hypothesis $H_0$ if the corresponding p-value is sufficiently small (usually smaller than 0.05). Many criticisms regarding p-values have arisen in the scientific literature. The main issue is that in general optimal p-values (based on likelihood ratio statistics) are not measures of evidence over the parameter space $\Theta$. Here, we propose an \emph{objective} measure of evidence for very general null hypotheses that satisfies logical requirements (i.e., operations on the subsets of $\Theta$) that are not met by p-values (e.g., it is a possibility measure). We study the proposed measure in the light of the abstract belief calculus formalism and we conclude that it can be used to establish objective states of belief on the subsets of $\Theta$. Based on its properties, we strongly recommend this measure as an additional summary of significance tests. At the end of the paper we give a short listing of possible open problems.