On the Performance of the Neyman Allocation with Small Pilots

TL;DR

This paper investigates the limitations of Neyman Allocation in experimental designs with small pilot studies, revealing it can increase variance under certain conditions and offering insights for better practice.

Contribution

The study provides an asymptotic analysis of Neyman Allocation with small pilots, highlighting scenarios where it performs poorly and suggesting alternative approaches.

Findings

Neyman Allocation can increase asymptotic variance with small pilots.

High kurtosis or homoskedasticity affects Neyman Allocation performance.

Empirical examples demonstrate practical relevance of the findings.

Abstract

The Neyman Allocation is used in many papers on experimental design, which typically assume that researchers have access to large pilot studies. This may be unrealistic. To understand the properties of the Neyman Allocation with small pilots, we study its behavior in an asymptotic framework that takes pilot size to be fixed even as the size of the main wave tends to infinity. Our analysis shows that the Neyman Allocation can lead to estimates of the ATE with higher asymptotic variance than with (non-adaptive) balanced randomization. In particular, this happens when the outcome variable is relatively homoskedastic with respect to treatment status or when it exhibits high kurtosis. We provide a series of empirical examples showing that such situations can arise in practice. Our results suggest that researchers with small pilots should not use the Neyman Allocation if they believe that outcomes are homoskedastic or heavy-tailed. Finally, we examine some potential methods for improving the finite sample performance of the FNA via simulations.