A Methodological Review of Simulation Studies Published in Pharmacoepidemiology and Drug Safety

TL;DR

This paper reviews simulation studies in pharmacoepidemiology to understand how they are designed and what they reveal about statistical methods.

Contribution

The paper provides a comprehensive review of simulation study practices in pharmacoepidemiology, highlighting trends and limitations.

Findings

Most simulation studies (81%) focused on comparative effectiveness/safety research.

Monte-Carlo simulations were used in 86% of the studies.

Only a few studies (4-3%) simulated time-varying effects or covariates.

Abstract

Simulation studies are used in pharmacoepidemiology for evaluating statistical methods in a controlled setting, whereby a known data‐generating mechanism allows evaluation of the performance of different approaches and assumptions. This study aimed to review simulation studies performed in pharmacoepidemiology.

We conducted a review of all papers published in the journal of Pharmacoepidemiology and Drug Safety (PDS) over the period 2017–2024. We extracted data on study characteristics and key simulation choices such as the type of data‐generating mechanism used, inferential methods tested and simulation size.

Among 42 simulation studies included, 34 (81%) were informing comparative effectiveness/safety studies. Twenty‐two studies (52%) used simulation in the context of a clinical condition, and 36 (86%) used Monte‐Carlo simulation. Inputs not derived from empirical data alone (n = 22, 52%) or in combination with real‐world data sources (n = 19, 45%) were most often used for data generation. The complexity of simulations was often relatively low: although 31 studies (74%) generated data based on other covariates, time‐dependent covariates (n = 3) and effects (n = 4) were rarely implemented. Bias was the most often used performance measure (n = 26, 62%), although notably 18 studies (43%) did not report uncertainty in the method.

Simulations contributed a relatively small number of articles (3.2% of 1320) to PDS over 2017–2024. Greater focus on evaluating methods and inferential approaches, using simulation studies that are appropriately complex given clinical realities, may be beneficial to the pharmacoepidemiology field.

Simulation studies comprised 3.2% (n = 42 of 1320) of articles reported in Pharmacoepidemiology and Drug Safety over 2017–2024. The type of simulations was mostly Monte‐Carlo‐based, where probability distributions were used to produce outcomes and/or covariates for analysis that have desired conditional or marginal properties.55% of studies were based on a specific clinical context that informed choices of the causal structure or inputs to the simulation.Studies rarely simulated time‐varying effects (n = 4) or covariates (n = 3), which may have implications for the transportability of the study's conclusions.The most common performance metric was bias (n = 26), but uncertainty was often not reported (n = 18).

Simulation studies comprised 3.2% (n = 42 of 1320) of articles reported in Pharmacoepidemiology and Drug Safety over 2017–2024. The type of simulations was mostly Monte‐Carlo‐based, where probability distributions were used to produce outcomes and/or covariates for analysis that have desired conditional or marginal properties.

55% of studies were based on a specific clinical context that informed choices of the causal structure or inputs to the simulation.

Studies rarely simulated time‐varying effects (n = 4) or covariates (n = 3), which may have implications for the transportability of the study's conclusions.

The most common performance metric was bias (n = 26), but uncertainty was often not reported (n = 18).

Simulation studies are tools that enable the performance evaluation of one or more statistical methods. Simulations often include several choices of parameters in their design and our paper aimed to summarise the specific practices in use within pharmacoepidemiology. This study reviewed all articles published in the journal Pharmacoepidemiology and Drug Safety over the period 2017–2024 which included simulation to test a statistical method. We found 42 such papers and described their key details with a summary of results. Studies typically used fully synthetic data generation and reported bias in the method using the true value of the estimate. Fifty‐five percent of studies incorporated knowledge of a drug or disease in the generation of the data to more closely match reality.

Figures