Beyond Arbitrary Replications: A Principled Approach to Simulation Design in Causal Inference

TL;DR

This paper introduces TISCA, a new method that uses statistical power analysis to determine the optimal number of simulation replications in causal inference studies, improving reproducibility and efficiency.

Contribution

TISCA provides a principled, statistically justified approach to choosing simulation replications, addressing arbitrary practices and enhancing reproducibility in causal inference research.

Findings

TISCA effectively reduces the number of simulations needed while maintaining statistical power.

Application to a case study shows TISCA's efficiency over traditional methods.

The bibliometric study highlights the heterogeneity in current simulation practices.

Abstract

Evaluation of novel treatment effect estimators frequently relies on simulation studies lacking formal statistical comparisons and using arbitrary numbers of replications ($J$). This hinders reproducibility and efficiency. We propose the Test-Informed Simulation Count Algorithm (TISCA) to address these shortcomings. TISCA integrates Welch's t-tests with power analysis, iteratively running simulations until a pre-specified power (e.g., 0.8) is achieved for detecting a user-defined minimum detectable effect size (MDE) at a given significance level ($\alpha$). This yields a statistically justified simulation count ($J$) and rigorous model comparisons. Our bibliometric study confirms the heterogeneity of current practices regarding $J$. A case study revisiting McJames et al. (2024) demonstrates TISCA identifies sufficient simulations ($J=500$ vs. original $J=1000$), saving computational resources while providing statistically sound evidence. TISCA promotes rigorous, efficient, and sustainable simulation practices in causal inference and beyond.