# Optimising error rates in programmes of pilot and definitive trials using Bayesian statistical decision theory

## TL;DR

This paper proposes a Bayesian decision-theoretic method to optimize error rates in pilot and definitive trials, showing that not testing for effectiveness in pilot trials can be inefficient.

## Contribution

A novel Bayesian decision-theoretic framework for optimizing pilot and definitive trial programs by balancing error rates and utility.

## Key findings

- The conventional approach of not testing for effectiveness in pilot trials can be sub-optimal.
- Optimal trial characteristics depend on parameters of the utility function, including risk attitude and costs.
- The method was applied to re-design the OK-Diabetes trial, demonstrating its practical utility.

## Abstract

Pilot trials are often conducted in advance of definitive trials to assess their feasibility and to inform their design. Although pilot trials typically collect primary endpoint data, preliminary tests of effectiveness have been discouraged given their typically low power. Power could be increased at the cost of a higher type I error rate, but there is little methodological guidance on how to determine the optimal balance between these operating characteristics. We consider a Bayesian decision-theoretic approach to this problem, introducing a utility function and defining an optimal pilot and definitive trial programme as that which maximises expected utility. We base utility on changes in average primary outcome, the cost of sampling, treatment costs, and the decision-maker’s attitude to risk. We apply this approach to re-design OK-Diabetes, a pilot trial of a complex intervention with a continuous primary outcome with known standard deviation. We then examine how optimal programme characteristics vary with the parameters of the utility function. We find that the conventional approach of not testing for effectiveness in pilot trials can be considerably sub-optimal.