Measuring representation in clinical trials: Simulations demonstrating how current methods fail in the context of precision medicine

TL;DR

This paper shows that current methods for measuring how well clinical trials represent diverse populations fail when applied to precision medicine, where treatments target specific subgroups.

Contribution

The paper introduces simulations to demonstrate how existing representativeness metrics break down when comparing to demographically distinct populations in precision medicine.

Findings

24%, 40%, and 32% of trials gave different representativeness results with 5 percentage point demographic shifts for sex, race, and ethnicity.

Larger demographic differences between registry and simulated populations led to worse performance of representativeness metrics.

Current methods are inaccurate for precision medicine contexts targeting specific subgroups.

Abstract

Clinical trial representativeness is vital for the evaluation of intervention performance and study generalizability. Current methods for evaluating representativeness are limited by the quality of the disease registry data used and may not appropriately evaluate studies aimed at precision cohorts. This study evaluates the sensitivity of existing methods for measuring study representation to differences between the population targeted by the clinical intervention and the closest available population in a patient registry. Using records for U.S.-based cancer clinical trials registered to ClinicalTrials.gov from 2017–2023 and the U.S. Cancer Statistics Public Use Database we calculated representativeness measures by comparing the demographic mix (based on sex, race, and ethnicity) in each clinical trial to the demographic mix for the same form of cancer in the U.S. Cancer Statistics database via exact Binomial tests. Then, the same tests were conducted comparing trial populations to simulated populations that were demographically different from the U.S. Cancer Statistics data by fixed percentages. The outcome of interest was whether the result of the test changed when the comparator population was different. For clinical trials reporting the sex, race, and ethnicity of participants, 24, 40, and 32 percent of studies (respectively) give different results when the difference between the registry population and simulated population is 5 percentage points. For all demographics, larger differences between the registry and the simulated populations were associated with worse metric performance. Analyses of clinical trial representativeness suffer from a large loss of accuracy in settings where treatments are targeted to demographically different subgroups. Studies of representation in the context of precision medicine should be interpreted with caution.

Figures