The paradoxical nature of easily improvable evidence

TL;DR

This paper explores how the perceived strength of scientific evidence should depend on the effort needed to improve it, revealing paradoxes where easily improvable evidence may falsely support claims.

Contribution

It introduces a novel perspective linking evidence persuasiveness to the effort required to improve it, supported by formal models demonstrating paradoxical effects.

Findings

Easily improvable evidence can paradoxically suggest a claim is false.

Publication of weak evidence may sometimes indicate the claim's falsehood.

Formal models reveal conditions under which evidence appears misleading.

Abstract

Established frameworks to understand problems with reproducibility in science begin with the relationship between our understanding of the prior probability of a claim and the statistical certainty that should be demanded of it, and explore the ways in which independent investigations, biases in study design and publication bias interact with these considerations. We propose a complementary perspective; namely, that to improve reproducibility in science, our interpretation of the persuasiveness of evidence (e.g., statistical significance thresholds) should be responsive to our understanding of the effort that would be required to improve that evidence. We will quantify this notion in some formal settings. Indeed, we will demonstrate that even simplistic models of evidence publication can exhibit an improvable evidence paradox, where the publication of easily improvable evidence in favor of a claim can best seen as evidence the claim is false.