A Not-So-Fundamental Limitation on Studying Complex Systems with Statistics: Comment on Rabin (2011)

TL;DR

The paper argues that the perceived limitations of statistical inference in complex systems are overstated, and that biases and selective reporting better explain phenomena like the 'Truth Wears Off' effect than fundamental statistical constraints.

Contribution

It challenges Rabin's claim of a fundamental limitation on studying complex systems with statistics, emphasizing the role of biases and reporting practices instead.

Findings

Biases and selective reporting explain the 'Truth Wears Off' effect.

Similar decline effects occur in simple physical systems.

No fundamental statistical limitation is necessary to explain these phenomena.

Abstract

Although living organisms are affected by many interrelated and unidentified variables, this complexity does not automatically impose a fundamental limitation on statistical inference. Nor need one invoke such complexity as an explanation of the "Truth Wears Off" or "decline" effect; similar "decline" effects occur with far simpler systems studied in physics. Selective reporting and publication bias, and scientists' biases in favour of reporting eye-catching results (in general) or conforming to others' results (in physics) better explain this feature of the "Truth Wears Off" effect than Rabin's suggested limitation on statistical inference.