Case Study: Evaluation of a meta-analysis of the association between soy protein and cardiovascular disease

TL;DR

This study critically examines a meta-analysis claiming soy protein benefits cardiovascular health, revealing potential biases, assumption violations, and lack of publication bias correction, thus challenging the reliability of its conclusions.

Contribution

It provides an independent re-evaluation of a meta-analysis using p-value plotting and bias detection techniques, highlighting issues in the original analysis.

Findings

Reported p-values may be biased due to multiple testing.

Meta-analysis assumptions are not satisfied in the data.

Publication bias is unaddressed in the original study.

Abstract

It is well-known that claims coming from observational studies most often fail to replicate. Experimental (randomized) trials, where conditions are under researcher control, have a high reputation and meta-analysis of experimental trials are considered the best possible evidence. Given the irreproducibility crisis, experiments lately are starting to be questioned. There is a need to know the reliability of claims coming from randomized trials. A case study is presented here independently examining a published meta-analysis of randomized trials claiming that soy protein intake improves cardiovascular health. Counting and p-value plotting techniques (standard p-value plot, p-value expectation plot, and volcano plot) are used. Counting (search space) analysis indicates that reported p-values from the meta-analysis could be biased low due to multiple testing and multiple modeling. Plotting techniques used to visualize the behavior of the data set used for meta-analysis suggest that statistics drawn from the base papers do not satisfy key assumptions of a random-effects meta-analysis. These assumptions include using unbiased statistics all drawn from the same population. Also, publication bias is unaddressed in the meta-analysis. The claim that soy protein intake should improve cardiovascular health is not supported by our analysis.