Between-trial heterogeneity in meta-analyses may be partially explained by reported design characteristics

TL;DR

This study explores how reported design biases like sequence generation, allocation concealment, and blinding contribute to heterogeneity in meta-analyses, suggesting bias adjustment might improve meta-analytic accuracy.

Contribution

It provides quantitative estimates of how much heterogeneity can be explained by specific risk of bias characteristics using Bayesian hierarchical models.

Findings

High/unclear risk of bias in sequence generation and blinding increases heterogeneity.

Approximately 37% of heterogeneity variance can be explained by bias-related factors.

Wide confidence intervals indicate imprecise estimates, limiting definitive conclusions.

Abstract

Objective: We investigated the associations between risk of bias judgments from Cochrane reviews for sequence generation, allocation concealment and blinding and between-trial heterogeneity. Study Design and Setting: Bayesian hierarchical models were fitted to binary data from 117 meta-analyses, to estimate the ratio {\lambda} by which heterogeneity changes for trials at high/unclear risk of bias, compared to trials at low risk of bias. We estimated the proportion of between-trial heterogeneity in each meta-analysis that could be explained by the bias associated with specific design characteristics. Results: Univariable analyses showed that heterogeneity variances were, on average, increased among trials at high/unclear risk of bias for sequence generation ({\lambda} 1.14, 95% interval: 0.57 to 2.30) and blinding ({\lambda} 1.74, 95% interval: 0.85 to 3.47). Trials at high/unclear risk of bias for allocation concealment were on average less heterogeneous ({\lambda} 0.75, 95% interval: 0.35 to 1.61). Multivariable analyses showed that a median of 37% (95% interval: 0% to 71%) heterogeneity variance could be explained by trials at high/unclear risk of bias for sequence generation, allocation concealment and/or blinding. All 95% intervals for changes in heterogeneity were wide and included the null of no difference. Conclusion: Our interpretation of the results is limited by imprecise estimates. There is some indication that between-trial heterogeneity could be partially explained by reported design characteristics, and hence adjustment for bias could potentially improve accuracy of meta-analysis results.