Can We Mathematically Spot Possible Manipulation of Results in Research Manuscripts Using Benford's Law?

TL;DR

This paper presents a semi-automatic method using an adaptive version of Benford's law to detect potential manipulation in research results from manuscripts, validated on datasets and recent economic journal articles.

Contribution

It introduces a novel application of an adaptive Benford's law approach to identify possible data manipulation in research manuscripts without needing raw data.

Findings

Predicted 79% of datasets accurately using the method.

Detected a 3% potential manipulation rate in recent economic papers.

Method offers a new tool for verifying research integrity.

Abstract

The reproducibility of academic research has long been a persistent issue, contradicting one of the fundamental principles of science. What is even more concerning is the increasing number of false claims found in academic manuscripts recently, casting doubt on the validity of reported results. In this paper, we utilize an adaptive version of Benford's law, a statistical phenomenon that describes the distribution of leading digits in naturally occurring datasets, to identify potential manipulation of results in research manuscripts, solely using the aggregated data presented in those manuscripts. Our methodology applies the principles of Benford's law to commonly employed analyses in academic manuscripts, thus, reducing the need for the raw data itself. To validate our approach, we employed 100 open-source datasets and successfully predicted 79% of them accurately using our rules. Additionally, we analyzed 100 manuscripts published in the last two years across ten prominent economic journals, with ten manuscripts randomly sampled from each journal. Our analysis predicted a 3% occurrence of result manipulation with a 96% confidence level. Our findings uncover disturbing inconsistencies in recent studies and offer a semi-automatic method for their detection.