Posterior Predictive P-values with Fisher Randomization Tests in Noncompliance Settings: Test Statistics vs Discrepancy Variables

TL;DR

This paper investigates the use of posterior predictive p-values with Fisher randomization tests in noncompliance settings, comparing test statistics and discrepancy variables, and examining the effects of modeling choices and misspecification on test validity and power.

Contribution

It provides a comprehensive comparison of discrepancy measures and classical test statistics in permutation tests for noncompliance, highlighting the impact of modeling assumptions and proposing more robust Bayesian approaches.

Findings

Modeling choices affect power and validity of tests.

Imputing compliance assuming the null reduces power; not doing so risks invalidity.

Permutation steps improve robustness of Bayesian tests to model misspecification.

Abstract

In randomized experiments with noncompliance, tests may focus on compliers rather than on the overall sample. Rubin (1998) put forth such a method, and argued that testing for the complier average causal effect and averaging permutation based p-values over the posterior distribution of the compliance status could increase power, as compared to general intent-to-treat tests. The general scheme is to repeatedly do a two-step process of imputing missing compliance statuses and conducting a permutation test with the completed data. In this paper, we explore this idea further, comparing the use of discrepancy measures, which depend on unknown but imputed parameters, to classical test statistics and exploring different approaches for imputing the unknown compliance statuses. We also examine consequences of model misspecification in the imputation step, and discuss to what extent this additional modeling undercuts the permutation test's model independence. We find that, especially for discrepancy measures, modeling choices can impact both power and validity. In particular, imputing missing compliance statuses assuming the null can radically reduce power, but not doing so can jeopardize validity. Fortunately, covariates predictive of compliance status can mitigate these results. Finally, we compare this overall approach to Bayesian model-based tests, that is tests that are directly derived from posterior credible intervals, under both correct and incorrect model specification. We find that adding the permutation step in an otherwise Bayesian approach improves robustness to model specification without substantial loss of power.