On the Logic (plus some history and philosophy) of Statistical Tests and Scientific Investigation

TL;DR

This paper explores the philosophical and historical foundations of statistical tests, analyzing four main paradigms and their impact on scientific reasoning, especially in psychology, highlighting both progress and pitfalls.

Contribution

It provides a comparative analysis of Fisher, Likelihood, Bayes, and Neyman-Pearson paradigms, emphasizing their conceptual differences and implications for scientific investigation.

Findings

Statistical testing has a straightforward mathematical basis.

Mainstream approaches have diverged from simple solutions, affecting scientific thinking.

Different paradigms influence the interpretation and application of statistical evidence.

Abstract

Every scientific endeavour consists of (at least) two components: A hypothesis on the one hand and data on the other. There is always a more or less abstract level - some theory, a set of concepts, certain relations of ideas - and a concrete level, i.e., empirical evidence, experiments or some observations which constitute matters of fact. The focus of this contribution is on elementary models connecting both levels that have been very popular in psychological research - statistical tests. Going from simple to complex we will examine four paradigms of statistical testing (Fisher, Likelihood, Bayes, Neyman & Pearson) and an elegant contemporary treatment. In a nutshell, testing is an easy problem that has a straightforward mathematical solution. However, it is rather surprising that the statistical mainstream has pursued a different line of argument. The application of the latter theory in psychology and other fields has brought some progress but has also impaired scientific thinking.