# Methodological quality of systematic intervention reviews on vaccination

## TL;DR

This study finds that most systematic reviews on vaccination are of low methodological quality, with issues like poor protocol registration and lack of conflict of interest management.

## Contribution

The study evaluates methodological quality of vaccination-related systematic reviews and identifies factors influencing their reliability.

## Key findings

- 110 out of 120 systematic reviews were rated as critically low-quality.
- Reviews published after 2017 and Cochrane reviews showed higher methodological quality.
- Only a quarter of reviews with conflicts of interest described how they managed them.

## Abstract

Systematic reviews (SRs) are pivotal in vaccination recommendation development. However, the reliability of their findings relies on methodological rigor. This study examines the methodological quality of vaccination-related SRs and aims to identify influencing factors.

We used the SYSVAC registry to draw 120 SRs on the efficacy or effectiveness of vaccination using a randomized block design. SRs published from 2011 to 2023 were considered. SR characteristics were extracted, and methodological quality was assessed by two reviewers using AMSTAR 2 (A Measurement Tool to Assess Systematic Reviews 2). SRs were scored between 0 and 16 and categorized as high, moderate, low, or critically low quality. Mann–Whitney U test, chi-squared test, Fisher’s exact test, and multivariable linear regression analysis were used to assess the influence of SR characteristics on AMSTAR 2 score. Secondary analysis of critically low-rated SRs was conducted to identify limitations in critical items.

Out of 120 SRs, 110 SRs were rated as of critically low-quality. The majority of critically low-rated SRs lacked the justification for excluding individual studies (n = 103, 93.6%) and protocol registration (n = 85, 77.3%). Median AMSTAR 2 score across all SRs was 10 (range 2–16). SRs published after 2017, Cochrane reviews, and SRs from authors with no conflicts of interest (CoI) had higher methodological quality than those published before 2017, non-Cochrane reviews, and SRs from authors with CoI (mean difference AMSTAR 2 score 2, 6, and 2, respectively; all p < 0.05). SRs published before 2017 had significant limitations in protocol registration, study selection in duplicate, and risk of bias assessment; and non-Cochrane reviews in protocol registration, justification for excluding individual studies, funding sources of studies, and impact of risk of bias assessment of individual studies (all p < 0.05). Management of conflicts was described only in a quarter of SRs with CoI (n = 9/43, 20.9%). The adjusted analysis showed that only publication year after 2017 and Cochrane review status had a significant independent relation on the AMSTAR 2 score (R2 = 0.26; adjusted R2 = 0.17).

The findings underscore the critical methodological shortcomings in vaccination-related SRs. Future efforts should prioritize adherence to established methodological standards and transparency, to enhance the impact of SRs in individual and decision-making.

The online version contains supplementary material available at 10.1186/s13643-025-03052-2.