Multiplicity adjustment approaches in publicly funded multi-arm trials: a comprehensive review of the National Institute for Health and Care Research (NIHR) Journals Library

TL;DR

This study reviews publicly funded multi-arm clinical trials in the UK and finds that most do not adjust for statistical multiplicity, which could increase false positive results.

Contribution

The study provides the first comprehensive review of multiplicity adjustment practices in publicly funded multi-arm trials in the UK.

Findings

Only 35% of reviewed trials adjusted for multiplicity in sample size calculations.

37% of trials made multiplicity adjustments in their statistical analysis.

Bonferroni was the most common adjustment method used.

Abstract

Parallel-group multi-arm trials are randomised controlled trials (RCTs) where participants are allocated to three or more concurrent treatment groups. Multiplicity occurs when several statistical tests are conducted within the same study. Statistical adjustments to the design and analysis of multi-arm trials can be used to control the study-wise type I error rate. There is no clear guidance or consensus on the necessity of multiplicity adjustment in multi-arm trials, nor on which methods are most appropriate. This comprehensive review aimed to investigate the design, analysis and reporting of publicly funded parallel-group multi-arm trials and to report the approach to multiplicity in these trials with respect to sample size and statistical analysis.

We searched the United Kingdom’s National Institute for Health and Care Research (NIHR) online Journals Library, from 1 January 1997 to 31 December 2024 for reports of multi-arm RCTs. Information on the trial characteristics, the sample size estimation and analysis of the primary outcome was extracted. Two researchers conducted the search and selected reports for inclusion. Data from each report was independently extracted by two reviewers, and any disagreement was resolved by discussion.

A total of 2452 reports, published online in the NIHR Journals Library, were screened for eligibility; 97 reports of multi-arm parallel-group trials met the inclusion criteria. Of these, 90 included the results of a multi-arm efficacy analysis. In the review, 35% (34/97) of the trials did adjust for multiplicity in the sample size calculation; in 84% (76/90), the potential between-arm comparisons were described in the methods, and 37% (33/90) made a multiplicity adjustment in the analysis. A further 86% (77/86) reported 95% confidence intervals. For the minority of multi-arm trials that did adjust for multiplicity, the most common adjustment method was Bonferroni.

The majority of the publicly funded multi-arm trials did not adjust for multiplicity in the sample size, statistical analysis, or estimation of confidence intervals. Researchers should follow the Consolidated Standards of Reporting Trials guidelines for multi-arm trials and clearly state in protocols and trial reports whether a multiplicity adjustment was made or provide a reason if no adjustment was made.

The online version contains supplementary material available at 10.1186/s13063-025-09324-5.

Figures