Can I trust this paper?
Andrey Anikin

TL;DR
This paper teaches researchers how to identify signs of flawed or unreliable empirical studies to improve scientific trustworthiness.
Contribution
The paper introduces four key problems in empirical research and practical methods to detect them.
Findings
Researchers should inspect author and journal profiles and request raw data to detect potential fraud.
Low precision of effect sizes and signs of data dredging indicate insufficient or incorrect data.
Unjustified conclusions often result from hidden confounds or overgeneralization.
Abstract
After a decade of data falsification scandals and replication failures in psychology and related empirical disciplines, there are urgent calls for open science and structural reform in the publishing industry. In the meantime, however, researchers need to learn how to recognize tell-tale signs of methodological and conceptual shortcomings that make a published claim suspect. I review four key problems and propose simple ways to detect them. First, the study may be fake; if in doubt, inspect the authors’ and journal’s profiles and request to see the raw data to check for inconsistencies. Second, there may be too little data; low precision of effect sizes is a clear warning sign of this. Third, the data may not be analyzed correctly; excessive flexibility in data analysis can be deduced from signs of data dredging and convoluted post hoc theorizing in the text, while violations of model…
Genes, proteins, chemicals, diseases, species, mutations and cell lines named across the full text — each resolved to its canonical identifier and authoritative record.
Click any figure to enlarge with its caption.
Figure 1
Figure 2
Figure 3Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMeta-analysis and systematic reviews · scientometrics and bibliometrics research · Explainable Artificial Intelligence (XAI)
