Fast Algorithms for Exact Confidence Intervals in Randomized Experiments with Binary Outcomes

TL;DR

This paper introduces efficient algorithms for constructing exact confidence intervals in binary outcome randomized experiments, significantly reducing computational effort while maintaining validity across various experimental designs.

Contribution

It presents novel algorithms that compute exact confidence intervals with logarithmic complexity for certain designs, and establishes optimality and efficiency bounds.

Findings

Exact confidence intervals can be computed with O(log n) tests under specific designs.

The algorithms are optimal in an information-theoretic sense.

The approach extends to general Bernoulli designs with O(n^2) tests.

Abstract

We construct exact confidence intervals for the average treatment effect in randomized experiments with binary outcomes using sequences of randomization tests. Our approach does not rely on large-sample approximations and is valid for all sample sizes. Under a balanced Bernoulli design or a matched-pairs design, we show that exact confidence intervals can be computed using only $O(\log n)$ randomization tests, yielding an exponential reduction in the number of tests compared to brute-force. We further prove an information-theoretic lower bound showing that this rate is optimal. In contrast, under balanced complete randomization, the most efficient known procedures require $O(n\log n)$ randomization tests (Aronow et al., 2023), establishing a sharp separation between these designs. In addition, we extend our algorithm to general Bernoulli designs using $O(n^2)$ randomization tests.