Convincing Evidence

TL;DR

This paper explores how statisticians and practitioners determine the credibility of statistical methods, highlighting reliance on personal experience, reputation, and anecdotal evidence rather than formal validation.

Contribution

It sheds light on the informal decision-making processes and social factors influencing the trust in statistical methods among practitioners.

Findings

Practitioners rely on personal experience and colleagues' attitudes.

Authorship and reputation influence method credibility.

Formal validation is less emphasized than anecdotal evidence.

Abstract

Textbooks on statistics emphasize care and precision, via concepts such as reliability and validity in measurement, random sampling and treatment assignment in data collection, and causal identification and bias in estimation. But how do researchers decide what to believe and what to trust when choosing which statistical methods to use? How do they decide the credibility of methods? Statisticians and statistical practitioners seem to rely on a sense of anecdotal evidence based on personal experience and on the attitudes of trusted colleagues. Authorship, reputation, and past experience are thus central to decisions about statistical procedures.