Reliability of meta-analysis of an association between ambient air quality and development of asthma later in life

TL;DR

This study critically evaluates the reliability of a meta-analysis linking early-life air pollution to later asthma, revealing biases, heterogeneity, and potential analysis manipulation in the base studies, questioning the validity of the claimed associations.

Contribution

It introduces a novel p-value plot analysis to assess heterogeneity and bias in the meta-analysis of air pollution and asthma, highlighting issues with multiple testing and data manipulation.

Findings

Large number of statistical tests in base papers suggest bias

Heterogeneity in NO2 results indicates mixture and potential manipulation

PM2.5 p-values show randomness, indicating no true association

Abstract

Claims from observational studies often fail to replicate. A study was undertaken to assess the reliability of cohort studies used in a highly cited meta-analysis of the association between ambient nitrogen dioxide, NO2, and fine particulate matter, PM2.5, concentrations early in life and development of asthma later in life. The numbers of statistical tests possible were estimated for 19 base papers considered for the meta-analysis. A p-value plot for NO2 and PM2.5 was constructed to evaluate effect heterogeneity of p-values used from the base papers. The numbers of statistical tests possible in the base papers were large - median 13,824, interquartile range 1,536-221,184; range 96-42M, in comparison to statistical test results presented. Statistical test results drawn from the base papers are unlikely to provide unbiased measures for meta-analysis. The p-value plot indicated that heterogeneity of the NO2 results across the base papers is consistent with a two-component mixture. First, it makes no sense to average across a mixture in meta-analysis. Second, the shape of the p-value plot for NO2 appears consistent with the possibility of analysis manipulation to obtain small p-values in several of the cohort studies. As for PM2.5, all corresponding p-values fall on a 45-degree line indicating complete randomness rather than a true association. Our interpretation of the meta-analysis is that the random p-values indicating no cause-effect associations are more plausible and that their meta-analysis will not likely replicate in the absence of bias. We conclude that claims made in the base papers used for meta-analysis are unreliable due to bias induced by multiple testing and multiple modelling, MTMM. We also show there is evidence that the heterogeneity across the base papers used for meta-analysis is more complex than simple sampling from a normal process.