Operational Dosage: Implications of Capacity Constraints for the Design and Interpretation of Experiments

TL;DR

This paper models how capacity constraints in service interventions affect experimental outcomes, revealing that treatment effects depend on capacity and sample size, with statistical power peaking at intermediate sample sizes.

Contribution

It introduces a queueing theory-based model of operational dosage, highlighting the impact of capacity constraints on causal inference and experiment design.

Findings

Treatment effects depend on capacity and sample size.

Statistical power peaks at intermediate sample sizes.

Capacity constraints can cause experiments to fail at scale.

Abstract

We study RCTs that evaluate the impact of service interventions, for example, teachers or advisors conducting proactive outreach to at-risk students, medical providers giving medication adherence support by calling or texting, or social workers that conduct home visits. A defining feature of service interventions is that they are delivered by a capacity-constrained resource -- teachers, healthcare providers, or social workers -- whose limited availability creates causal inference complications. Because participants share a finite service capacity, adding more participants can reduce the timeliness or intensity of the service that others receive, introducing interference across participants. This generates hidden variation in the treatment itself, which we term operational dosage. We provide a mathematical model of service interventions using techniques from queueing theory and study the impact of capacity constraints on experimental outcomes. Our main insight is that treatment effects are both capacity- and sample-size-dependent, as well as decreasing in sample size once a critical threshold is exceeded. Interestingly, an implication is that statistical power of service intervention RCTs peaks at intermediate sample sizes -- directly contradicting conventional power calculations that assume monotonically increasing power with sample size. We instantiate our insights using simulations calibrated to a real-world trial evaluating a behavioral health intervention for tuberculosis patients in Kenya. Our simulation results suggest that a trial with high service capacity but limited sample size can obtain the same statistical power as a trial with lower service capacity but large sample size. Taken together, our results highlight the importance of capacity selection in experiment design and provide a mechanism for why experiments may fail to replicate or perform at scale.