Beyond p values: practical methods for analyzing uncertainty in research

TL;DR

This paper discusses practical methods beyond p values for assessing the certainty of research findings, emphasizing confidence intervals and estimated probabilities for hypotheses, including Bayesian interpretations.

Contribution

It introduces and compares approaches like confidence intervals and estimated hypothesis probabilities as alternatives to p values for uncertainty analysis.

Findings

Confidence intervals provide a range for numerical results.

Estimated probabilities offer direct certainty measures for hypotheses.

Methods can be applied using common statistical software like SPSS.

Abstract

This article explains, and discusses the merits of, three approaches for analyzing the certainty with which statistical results can be extrapolated beyond the data gathered. Sometimes it may be possible to use more than one of these approaches. (1) If there is an exact null hypothesis which is credible and interesting (usually not the case), researchers should cite a p value (significance level), although jargon is best avoided. (2) If the research result is a numerical value, researchers should cite a confidence interval. (3) If there are one or more hypotheses of interest, it may be possible to adapt the methods used for confidence intervals to derive an "estimated probability" for each. Under certain circumstances these could be interpreted as Bayesian posterior probabilities. These estimated probabilities can easily be worked out from the p values and confidence intervals produced by packages such as SPSS. Estimating probabilities for hypotheses means researchers can give a direct answer to the question "How certain can we be that this hypothesis is right?".