Estimates of heterogeneity (I2) can be biased in small meta-analyses

TL;DR

This paper demonstrates that the common estimator of heterogeneity (I2) in meta-analyses is biased, especially in small studies, and recommends cautious interpretation and the use of confidence intervals in such cases.

Contribution

It reveals the bias in I2 estimation in small meta-analyses and suggests using confidence intervals for more reliable interpretation.

Findings

Bias in I2 estimator increases with fewer studies.

Estimated I2 can be substantially inflated in small meta-analyses.

Confidence intervals are recommended over point estimates in small samples.

Abstract

In meta-analysis, the fraction of variance that is due to heterogeneity is known as I2. We show that the usual estimator of I2 is biased. The bias is largest when a meta-analysis has few studies and little heterogeneity. For example, with 7 studies and the true value of I2 at 0, the average estimate of I2 is .124. Estimates of I2 should be interpreted cautiously when the meta-analysis is small and the null hypothesis of homogeneity (I2=0) has not been rejected. In small meta-analyses, confidence intervals may be preferable to point estimates for I2.