Misuse of the sign test in narrative synthesis of evidence

TL;DR

This paper critiques the misuse of the sign test in narrative synthesis, highlighting its inappropriateness when only effect direction is available, and discusses implications for research conclusions.

Contribution

It clarifies the limitations of applying the sign test in narrative synthesis and warns against its misuse in evidence evaluation.

Findings

Sign test is inappropriate for effect direction data.

Misuse can lead to incorrect hypothesis testing.

Highlights need for suitable statistical methods in narrative synthesis.

Abstract

In narrative synthesis of evidence, it can be the case that the only quantitative measures available concerning the efficacy of an intervention is the direction of the effect, i.e. whether it is positive or negative. In such situations, the sign test has been proposed in the literature and in recent Cochrane guidelines as a way to test whether the proportion of positive effects is favourable. I argue that the sign test is inappropriate in this context as the data are not generated according to the Binomial distribution it employs. I demonstrate possible consequences for both hypothesis testing and estimation via hypothetical examples.