A general characterization of optimal tie-breaker designs

TL;DR

This paper characterizes optimal tie-breaker designs balancing statistical information gain and short-term treatment assignment benefits, providing sharp existence and uniqueness results and demonstrating improvements over previous designs with real-world data.

Contribution

It introduces a general framework for optimal tie-breaker designs with non-decreasing treatment probabilities, including existence, uniqueness, and practical implementation insights.

Findings

Optimal designs are constant below and above a single discontinuity.

Designs improve upon previous three-level tie-breaker designs in non-symmetric cases.

Application to Head Start data demonstrates practical benefits.

Abstract

Tie-breaker designs trade off a statistical design objective with short-term gain from preferentially assigning a binary treatment to those with high values of a running variable $x$. The design objective is any continuous function of the expected information matrix in a two-line regression model, and short-term gain is expressed as the covariance between the running variable and the treatment indicator. We investigate how to specify design functions indicating treatment probabilities as a function of $x$ to optimize these competing objectives, under external constraints on the number of subjects receiving treatment. Our results include sharp existence and uniqueness guarantees, while accommodating the ethically appealing requirement that treatment probabilities are non-decreasing in $x$. Under such a constraint, there always exists an optimal design function that is constant below and above a single discontinuity. When the running variable distribution is not symmetric or the fraction of subjects receiving the treatment is not $1/2$, our optimal designs improve upon a $D$-optimality objective without sacrificing short-term gain, compared to the three level tie-breaker designs of Owen and Varian (2020) that fix treatment probabilities at $0$, $1/2$, and $1$. We illustrate our optimal designs with data from Head Start, an early childhood government intervention program.