Standard meta-analysis methods are not robust

TL;DR

This paper highlights the lack of robustness in standard meta-analysis methods, especially when faced with multiple testing issues or fraudulent data, risking unreliable combined results.

Contribution

It critically examines the vulnerabilities of current meta-analysis techniques and proposes methods to evaluate the integrity of base studies and the reliability of combined results.

Findings

Standard meta-analysis methods are sensitive to extreme or fraudulent data.

Multiple testing and data fabrication can bias meta-analysis outcomes.

Proposed evaluation methods can help identify unreliable meta-analytic results.

Abstract

P values or risk ratios from multiple, independent studies, observational or randomized, can be computationally combined to provide an overall assessment of a research question in meta-analysis. There is a need to examine the reliability of these methods of combination. It is typical in observational studies to statistically test many questions and not correct the analysis results for multiple testing or multiple modeling, MTMM. The same problem can happen for randomized, experimental trials. There is the additional problem that some of the base studies may be using fabricated or fraudulent data. If there is no attention to MTMM or fraud in the base studies, there is no guarantee that the results to be combined are unbiased, the key requirement for the valid combining of results. We note that methods of combination are not robust; even one extreme base study value can overwhelm standard methods of combination. It is possible that multiple, extreme (MTMM or fraudulent) results can feed from the base studies to bias the combined result. A meta-analysis of observational (or even randomized studies) may not be reliable. Examples are given along with some methods to evaluate existing base studies and meta-analysis studies.