Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects

TL;DR

This paper introduces a simple two-stage data-driven RCT design that efficiently prunes low-impact treatments and provides high-probability guarantees on treatment effects, improving resource use and treatment effect estimation.

Contribution

It develops a novel two-stage RCT framework with a data-driven screening process and theoretical guarantees, suitable for scenarios with limited adaptivity.

Findings

Two-stage designs outperform single-stage approaches in treatment effect estimation.

The method effectively incorporates prior domain knowledge.

Sample splitting enables practical implementation of the optimal designs.

Abstract

Randomized controlled trials (RCTs) can be used to generate guarantees on treatment effects. However, RCTs often spend unnecessary resources exploring sub-optimal treatments, which can reduce the power of treatment guarantees. To address these concerns, we develop a two-stage RCT where, first on a data-driven screening stage, we prune low-impact treatments, while in the second stage, we develop high probability lower bounds on the treatment effect. Unlike existing adaptive RCT frameworks, our method is simple enough to be implemented in scenarios with limited adaptivity. We derive optimal designs for two-stage RCTs and demonstrate how we can implement such designs through sample splitting. Empirically, we demonstrate that two-stage designs improve upon single-stage approaches, especially in scenarios where domain knowledge is available in the form of a prior. Our work is thus, a simple, yet effective, method to estimate high probablility certificates for high performant treatment effects on a RCT.