More lessons from the six box toy experiment

TL;DR

This paper analyzes long sequences of probabilistic inference in a toy experiment, demonstrating the effectiveness of probability-based methods over simple rules and highlighting the irrelevance of small observed probabilities for inference.

Contribution

It provides a detailed analysis of simulated sequences, illustrating the strengths of probabilistic inference and challenging common assumptions about small probabilities in hypothesis testing.

Findings

Probability-based inference outperforms simple rules in the toy experiment.

Small observed probabilities are shown to be irrelevant for inference.

The analysis supports the philosophical foundation of probabilistic methods.

Abstract

Following a paper in which the fundamental aspects of probabilistic inference were introduced by means of a toy experiment, details of the analysis of simulated long sequences of extractions are shown here. In fact, the striking performance of probability-based inference and forecasting, compared to those obtained by simple `rules', might impress those practitioners who are usually underwhelmed by the philosophical foundation of the different methods. The analysis of the sequences also shows how the smallness of the probability of what has been actually observed, given the hypotheses of interest, is irrelevant for the purpose of inference.