Differences Among Noninformative Stopping Rules Are Often Relevant to Bayesian Decisions

TL;DR

This paper explores how differences in noninformative stopping rules impact classical and Bayesian statistical decisions, showing that Bayesian principles can clarify classical practices in scientific experiments.

Contribution

It demonstrates that Bayesian and classical ideas about stopping rules are mutually supportive in certain scientific contexts, clarifying their relationship.

Findings

Bayesian principles support classical penalization of biased stopping rules

Support for penalization is narrower under Bayesian than classical principles

Classical and Bayesian approaches align in specific experimental scenarios

Abstract

L.J. Savage once hoped to show that "the superficially incompatible systems of ideas associated on the one hand with [subjective Bayesianism] and on the other hand with [classical statistics]...lend each other mutual support and clarification." By 1972, however, he had largely "lost faith in the devices" of classical statistics. One aspect of those "devices" that he found objectionable is that differences among the "stopping rules" that are used to decide when to end an experiment which are "noninformative" from a Bayesian perspective can affect decisions made using a classical approach. Two experiments that produce the same data using different stopping rules seem to differ only in the intentions of the experimenters regarding whether or not they would have carried on if the data had been different, which seem irrelevant to the evidential import of the data and thus to facts about what actions the data warrant. I argue that classical and Bayesian ideas about stopping rules do in fact "lend each other" the kind of "mutual support and clarification" that Savage had originally hoped to find. They do so in a kind of case that is common in scientific practice, in which those who design an experiment have different interests from those who will make decisions in light of its results. I show that, in cases of this kind, Bayesian principles provide qualified support for the classical statistical practice of "penalizing" "biased" stopping rules. However, they require this practice in a narrower range of circumstances than classical principles do, and for different reasons. I argue that classical arguments for this practice are compelling in precisely the class of cases in which Bayesian principles also require it, and thus that we should regard Bayesian principles as clarifying classical statistical ideas about stopping rules rather than the reverse.