IQ: Intrinsic measure for quantifying the heterogeneity in meta-analysis

TL;DR

This paper critiques the widely used $I^2$ heterogeneity measure in meta-analysis, introduces a new intrinsic measure called IQ, and demonstrates its advantages through theoretical analysis, simulations, and real data applications.

Contribution

The paper identifies flaws in the $I^2$ statistic, proposes the IQ measure as a more reliable alternative, and develops an optimal estimator for practical use.

Findings

IQ provides nearly unbiased heterogeneity estimates.

IQ is independent of study sample sizes.

Simulations and real data confirm IQ's effectiveness.

Abstract

Quantifying the heterogeneity is an important issue in meta-analysis, and among the existing measures, the $I^2$ statistic is the most commonly used measure in the literature. In this paper, we show that the $I^2$ statistic was, in fact, defined as problematic or even completely wrong from the very beginning. To confirm this statement, we first present a motivating example to show that the $I^2$ statistic is heavily dependent on the study sample sizes, and consequently it may yield contradictory results for the amount of heterogeneity. Moreover, by drawing a connection between ANOVA and meta-analysis, the $I^2$ statistic is shown to have, mistakenly, applied the sampling errors of the estimators rather than the variances of the study populations. Inspired by this, we introduce an Intrinsic measure for Quantifying the heterogeneity in meta-analysis, and meanwhile study its statistical properties to clarify why it is superior to the existing measures. We further propose an optimal estimator, referred to as the IQ statistic, for the new measure of heterogeneity that can be readily applied in meta-analysis. Simulations and real data analysis demonstrate that the IQ statistic provides a nearly unbiased estimate of the true heterogeneity and it is also independent of the study sample sizes.