The ultimate issue error in scientific inference: mistaking parameters for hypotheses

TL;DR

This paper highlights the widespread mistake of confusing parameter probabilities with hypothesis probabilities in scientific inference, advocating for the Weight of Evidence approach to improve accuracy and transparency.

Contribution

It introduces the Weight of Evidence method as a superior alternative to NHST, demonstrating its application in assessing vitamin D and COVID-19 risk with better contextual integration.

Findings

WoE quantifies support for hypotheses considering biases and confounders

Applying WoE improves inference transparency and reproducibility

Highlights the need to move beyond traditional NHST methods

Abstract

Statistical inference often conflates the probability of a parameter with the probability of a hypothesis, a critical misunderstanding termed the ultimate issue error. This error is pervasive across the social, biological, and medical sciences, where null hypothesis significance testing (NHST) is mistakenly understood to be testing hypotheses rather than evaluating parameter estimates. Here, we advocate for using the Weight of Evidence (WoE) approach, which integrates quantitative data with qualitative background information for more accurate and transparent inference. Through a detailed example involving the relationship between vitamin D (25-hydroxy vitamin D) levels and COVID-19 risk, we demonstrate how WoE quantifies support for hypotheses while accounting for study design biases, power, and confounding factors. These findings emphasise the necessity of combining statistical metrics with contextual evaluation. This offers a structured framework to enhance reproducibility, reduce false interpretations, and foster robust scientific conclusions across disciplines.