Harmonizing Fully Optimal Designs with Classic Randomization in Fixed Trial Experiments

TL;DR

This paper compares optimized fixed-sample trial designs with traditional randomization, analyzing their benefits and costs, and suggests that a hybrid approach may be optimal until further methods are developed.

Contribution

It introduces a framework to understand the trade-offs between optimal and random designs in fixed trials, highlighting that pure optimization may not always outperform randomization.

Findings

Randomization outperforms optimization under Efron's worst-case criterion.

Optimal designs are intermediate between pure randomization and full optimization.

Further research is needed to develop practical procedures for optimal design selection.

Abstract

There is a movement in design of experiments away from the classic randomization put forward by Fisher, Cochran and others to one based on optimization. In fixed-sample trials comparing two groups, measurements of subjects are known in advance and subjects can be divided optimally into two groups based on a criterion of homogeneity or "imbalance" between the two groups. These designs are far from random. This paper seeks to understand the benefits and the costs over classic randomization in the context of different performance criterions such as Efron's worst-case analysis. In the criterion that we motivate, randomization beats optimization. However, the optimal design is shown to lie between these two extremes. Much-needed further work will provide a procedure to find this optimal designs in different scenarios in practice. Until then, it is best to randomize.