How sure are we? Two approaches to statistical inference

TL;DR

This paper compares two statistical inference approaches—null hypothesis testing and a Bayesian-like method—using simple, simulation-based explanations suitable for non-experts, to better understand the confidence in statistical conclusions.

Contribution

It introduces a minimalist, simulation-based approach to statistical inference, clarifying both traditional and Bayesian methods for non-specialists.

Findings

Approach 1 (null hypothesis testing) has significant limitations.

Approach 2 (Bayesian-like) offers probability estimates of hypotheses.

Simulation methods can effectively illustrate statistical concepts without complex distributions.

Abstract

Suppose you are told that taking a statin will reduce your risk of a heart attack or stroke by 3% in the next ten years, or that women have better emotional intelligence than men. You may wonder how accurate the 3% is, or how confident we should be about the assertion about women's emotional intelligence, bearing in mind that these conclusions are only based on samples of data? My aim here is to present two statistical approaches to questions like these. Approach 1 is often called null hypothesis testing but I prefer the phrase "baseline hypothesis": this is the standard approach in many areas of inquiry but is fraught with problems. Approach 2 can be viewed as a generalisation of the idea of confidence intervals, or as the application of Bayes' theorem. Unlike Approach 1, Approach 2 provides a tentative estimate of the probability of hypotheses of interest. For both approaches, I explain, from first principles, building only on "common sense" statistical concepts like averages and randomness, both how to derive answers, and the rationale behind the answers. This is achieved by using computer simulation methods (resampling and bootstrapping using a spreadsheet available on the web) which avoid the use of probability distributions (t, normal, etc). Such a minimalist, but reasonably rigorous, analysis is particularly useful in a discipline like statistics which is widely used by people who are not specialists. My intended audience includes both statisticians, and users of statistical methods who are not statistical experts.