Impact of matrix-construction assumptions on quantitative overlap assessment in overviews: A meta-research study

TL;DR

This study examines how different assumptions in constructing evidence matrices affect overlap assessments in systematic reviews, showing significant variability in results.

Contribution

The study introduces preliminary guidance for transparently reporting matrix-construction assumptions to improve CCA accuracy and reproducibility.

Findings

CCA values varied from 1.2% to 13.5% with the overall method and 0.0% to 15.7% with the pairwise method.

Only 11.9% of overviews adhered to a specific reporting guideline, and 44.0% did not address overlap at all.

Variability in CCA highlights the need for transparent reporting of matrix-construction assumptions.

Abstract

Overlap of primary studies among multiple systematic reviews (SRs) is a major challenge when conducting overviews. The corrected covered area (CCA) is a metric computed from a matrix of evidence that quantifies overlap. Therefore, the assumptions used to generate the matrix may significantly affect the CCA. We aim to explore how these varying assumptions influence CCA calculations. We searched two databases for intervention-focused overviews published during 2023. Two reviewers conducted study selection and data extraction. We extracted overview characteristics and methods to handle overlap. For seven sampled overviews, we calculated overall and pairwise CCA across 16 scenarios, representing four matrix-construction assumptions. Of 193 included overviews, only 23 (11.9%) adhered to an overview-specific reporting guideline (e.g. PRIOR). Eighty-five (44.0%) did not address overlap; 14 (7.3%) only mentioned it in the discussion; and 94 (48.7%) incorporated it into methods or results (38 using CCA). Among the seven sampled overviews, CCA values varied depending on matrix-construction assumptions, ranging from 1.2% to 13.5% with the overall method and 0.0% to 15.7% with the pairwise method. CCA values may vary depending on the assumptions made during matrix construction, including scope, treatment of structural missingness, and handling of publication threads. This variability calls into question the uncritical use of current CCA thresholds and underscores the need for overview authors to report both overall and pairwise CCA calculations. Our preliminary guidance for transparently reporting matrix-construction assumptions may improve the accuracy and reproducibility of CCA assessments.

Figures