Treatment effect validation via a permutation test in Stata

TL;DR

This paper introduces a permutation test procedure for validating treatment effects in experimental studies with baseline imbalance and attrition, implemented through a new Stata command, and evaluates its performance via simulation.

Contribution

It presents a new Stata command for permutation testing in treatment effect analysis and assesses its validity and power through Monte Carlo simulations.

Findings

The permutation test maintains appropriate size under baseline imbalance.

The test exhibits good power in detecting true treatment effects.

Finite-sample performance is validated through simulation studies.

Abstract

In this paper we describe the testing procedure for assessing the statistical significance of treatment effect under the experimental conditions of baseline imbalance across covariates and attrition from the survey, using the permutation tests proposed by Freedman and Lane (1983) and Romano and Wolf (2016). We discuss the testing procedure for these hypotheses based on a linear regression model and introduce the new Stata command [R] permtest for the implementation of the permutation test in Stata. Moreover, we investigate the finite-sample performance as well as the statistical validity of the test with a Monte Carlo simulation study in which we examine the empirical size and power properties under the conditions of baseline imbalance and attrition for a fixed number of permutation steps.