Statistical Evidence Measured on a Properly Calibrated Scale Across Nested and Non-nested Hypothesis Comparisons

TL;DR

This paper advances a thermodynamics-inspired framework for measuring statistical evidence, extending it to various hypothesis comparisons and developing calibration principles to make it practical for broad use.

Contribution

It generalizes the evidence measurement framework beyond simple cases using a Van der Waals-like equation and develops calibration principles related to degrees of freedom.

Findings

Extended the framework to multiple hypothesis comparisons

Introduced a Van der Waals-like extension of the ideal gas equation

Developed calibration principles linked to degrees of freedom

Abstract

Statistical modeling is often used to measure the strength of evidence for or against hypotheses on given data. We have previously proposed an information-dynamic framework in support of a properly calibrated measurement scale for statistical evidence, borrowing some mathematics from thermodynamics, and showing how an evidential analogue of the ideal gas equation of state could be used to measure evidence for a one-sided binomial hypothesis comparison (coin is fair versus coin is biased towards heads). Here we take three important steps forward in generalizing the framework beyond this simple example. We (1) extend the scope of application to other forms of hypothesis comparison in the binomial setting; (2) show that doing so requires only the original ideal gas equation plus one simple extension, which has the form of the Van der Waals equation; (3) begin to develop the principles required to resolve a key constant, which enables us to calibrate the measurement scale across applications, and which we find to be related to the familiar statistical concept of degrees of freedom. This paper thus moves our information-dynamic theory substantially closer to the goal of producing a practical, properly calibrated measure of statistical evidence for use in general applications.