A Simple Way to Deal with Cherry-picking

TL;DR

This paper addresses the problem of cherry-picking in hypothesis testing, especially in machine learning, proposing a post-reporting verification method to reduce false discoveries caused by selection bias.

Contribution

It introduces a simple post-reporting verification approach to mitigate bias in reported results, supported by theoretical analysis and experimental validation.

Findings

Post-reporting verification reduces false discoveries.

The method is effective on synthetic and real datasets.

Selection bias can lead to false claims of innovation.

Abstract

Statistical hypothesis testing serves as statistical evidence for scientific innovation. However, if the reported results are intentionally biased, hypothesis testing no longer controls the rate of false discovery. In particular, we study such selection bias in machine learning models where the reporter is motivated to promote an algorithmic innovation. When the number of possible configurations (e.g., datasets) is large, we show that the reporter can falsely report an innovation even if there is no improvement at all. We propose a `post-reporting' solution to this issue where the bias of the reported results is verified by another set of results. The theoretical findings are supported by experimental results with synthetic and real-world datasets.