Statistical Decision Properties of Imprecise Trials Assessing COVID-19 Drugs

TL;DR

This paper evaluates decision-making criteria in small-sample COVID-19 drug trials, finding that the empirical success rule outperforms traditional hypothesis testing in near-optimality.

Contribution

It introduces the use of near-optimality to assess decision criteria and demonstrates the superiority of the empirical success rule over hypothesis tests in COVID-19 trials.

Findings

Empirical success rule yields more optimal treatment decisions.

Traditional hypothesis tests often lead to sub-optimal choices.

The approach improves decision quality in small-sample trials.

Abstract

As the COVID-19 pandemic progresses, researchers are reporting findings of randomized trials comparing standard care with care augmented by experimental drugs. The trials have small sample sizes, so estimates of treatment effects are imprecise. Seeing imprecision, clinicians reading research articles may find it difficult to decide when to treat patients with experimental drugs. Whatever decision criterion one uses, there is always some probability that random variation in trial outcomes will lead to prescribing sub-optimal treatments. A conventional practice when comparing standard care and an innovation is to choose the innovation only if the estimated treatment effect is positive and statistically significant. This practice defers to standard care as the status quo. To evaluate decision criteria, we use the concept of near-optimality, which jointly considers the probability and magnitude of decision errors. An appealing decision criterion from this perspective is the empirical success rule, which chooses the treatment with the highest observed average patient outcome in the trial. Considering the design of recent and ongoing COVID-19 trials, we show that the empirical success rule yields treatment results that are much closer to optimal than those generated by prevailing decision criteria based on hypothesis tests.