Optimising the Trade-Off Between Type I and Type II Errors: A Review and Extensions

TL;DR

This paper reviews methods for optimizing the balance between Type I and Type II errors in clinical studies, considering costs and prior beliefs, and extends these approaches to more complex hypotheses, linking to study success probabilities.

Contribution

It introduces extensions to existing trade-off optimization methods for Type I and II errors, incorporating composite hypotheses and linking to clinical success probabilities.

Findings

Proposes cost-aware error rate optimization methods.

Extends trade-off analysis to composite hypotheses.

Links error trade-offs to Probability of Success in studies.

Abstract

In clinical studies upon which decisions are based there are two types of errors that can be made: a type I error arises when the decision is taken to declare a positive outcome when the truth is in fact negative, and a type II error arises when the decision is taken to declare a negative outcome when the truth is in fact positive. Commonly the primary analysis of such a study entails a two-sided hypothesis test with a type I error rate of 5% and the study is designed to have a sufficiently low type II error rate, for example 10% or 20%. These values are arbitrary and often do not reflect the clinical, or regulatory, context of the study and ignore both the relative costs of making either type of error and the sponsor's prior belief that the drug is superior to either placebo, or a standard of care if relevant. This simplistic approach has recently been challenged by numerous authors both from a frequentist and Bayesian perspective since when resources are constrained there will be a need to consider a trade-off between type I and type II errors. In this paper we review proposals to utilise the trade-off by formally acknowledging the costs to optimise the choice of error rates for simple, point null and alternative hypotheses and extend the results to composite, or interval hypotheses, showing links to the Probability of Success of a clinical study.