On the Use of Design-Based Simulations

TL;DR

This paper critically examines the effectiveness of design-based simulations in assessing inference validity, especially in shift-share designs, and proposes alternative methods to improve their reliability.

Contribution

It identifies limitations of standard simulations in shift-share contexts and introduces new simulation designs that better reflect the true data-generating process.

Findings

Standard simulations may conflate treatment effects with error dependence.

Proposed alternative simulations avoid misalignment with the true process.

The usefulness of simulations depends on their alignment with the true data-generating process.

Abstract

Design-based simulations - procedures that hold realized outcomes fixed and generate variation by resampling treatment assignment or shocks - are widely used in both methodological and applied work to assess inference procedures. This paper studies the extent to which such simulations are informative about inference validity. Focusing on shift-share designs, we show that standard simulations that fix outcomes and resample shocks may rely on a data-generating process that is not aligned with the true one. In particular, these simulations confound true treatment effects with error dependence, potentially overstating inference distortions due to spatial correlation. We propose alternative simulation designs that circumvent this problem and illustrate their use in prominent empirical applications. Our results highlight that the usefulness of design-based simulations depends critically on how closely the simulated data-generating process aligns with the true one.