Evaluating A Key Instrumental Variable Assumption Using Randomization Tests

TL;DR

This paper introduces a new falsification test for instrumental variable validity that compares observed covariate balance to what would be expected under randomization, providing a global and assumption-free assessment.

Contribution

It proposes an innovative, non-parametric falsification test that evaluates IV validity by comparing covariate balance to a randomized benchmark, applicable across all covariates.

Findings

The test can assess if the IV is closer to randomization than the exposure.

It allows for graphical and global balance assessments.

Demonstrated on ICU bed availability as an IV in healthcare data.

Abstract

Instrumental variable (IV) analyses are becoming common in health services research and epidemiology. Most IV analyses use naturally occurring instruments, such as distance to a hospital. In these analyses, investigators must assume the instrument is as-if randomly assigned. This assumption cannot be tested directly, but it can be falsified. Most falsification tests in the literature compare relative prevalence or bias in observed covariates between the instrument and the exposure. These tests require investigators to make a covariate-by-covariate judgment about the validity of the IV design. Often, only some of the covariates are well-balanced, making it unclear if as-if randomization can be assumed for the instrument across all covariates. We propose an alternative falsification test that compares IV balance or bias to the balance or bias that would have been produced under randomization. A key advantage of our test is that it allows for global balance measures as well as easily interpretable graphical comparisons. Furthermore, our test does not rely on any parametric assumptions and can be used to validly assess if the instrument is significantly closer to being as-if randomized than the exposure. We demonstrate our approach on a recent IV application that uses bed availability in the intensive care unit (ICU) as an instrument for admission to the ICU.

Tables

Table 1. Table 1 : The four possible cases (and our corresponding recommendation) when using our randomization test to assess the hypothesis of as-if randomization ( H 0 subscript 𝐻 0 H_{0} ) for 𝑫 𝑫 \bm{D} and 𝒁 𝒁 \bm{Z} .

	Exposure ($𝑫$)	Instrument ($𝒁$)	Recommendation
Case 1	Reject $H_0$	Fail to Reject $H_0$	Use IV analysis
Case 2	Fail to Reject $H_0$	Reject $H_0$	Reject IV analysis
Case 3	Fail to Reject $H_0$	Fail to Reject $H_0$	Use IV analysis or exposure analysis
Case 4	Reject $H_0$	Reject $H_0$	Compare balance or bias of $𝑫$ and $𝒁$

Equations

$$D=D(Z)=ZD(1)+(1-Z)D(0),$$

$$Y=Y(Z,D)=Y\left(Z,D(Z)\right)=ZY\left(1,D(1)\right)+(1-Z)Y\left(0,D(0)\right).$$

$$P_{cr}(\mathbf{Z}=\mathbf{z}|\mathbf{X})$$

$$p=\frac{1+{\sum }_{m=1}^M\mathbb{I}(|t({\mathbf{z}}^{(m)},\mathbf{X})|\ge |t^{obs}|)}{M+1},\text{where tobs≡t(Zobs,X)}$$

$$\underset{\text{covariate prevalence difference}}{\underbrace{{\overline{\mathbf{X}}}_{Z=1}-{\overline{\mathbf{X}}}_{Z=0}}}\text{or}\underset{\text{IV bias}}{\underbrace{\frac{{\overline{\mathbf{X}}}_{Z=1}-{\overline{\mathbf{X}}}_{Z=0}}{{\overline{\mathbf{D}}}_{Z=1}-{\overline{\mathbf{D}}}_{Z=0}}}}$$

$$M_Z$$

$${\left(\frac{{\overline{\mathbf{X}}}_{Z=1}-{\overline{\mathbf{X}}}_{Z=0}}{{\overline{\mathbf{D}}}_{Z=1}-{\overline{\mathbf{D}}}_{Z=0}}\right)}^T{\left[\text{cov}\left(\frac{{\overline{\mathbf{X}}}_{Z=1}-{\overline{\mathbf{X}}}_{Z=0}}{{\overline{\mathbf{D}}}_{Z=1}-{\overline{\mathbf{D}}}_{Z=0}}\right)\right]}^{-1}\left(\frac{{\overline{\mathbf{X}}}_{Z=1}-{\overline{\mathbf{X}}}_{Z=0}}{{\overline{\mathbf{D}}}_{Z=1}-{\overline{\mathbf{D}}}_{Z=0}}\right)$$

$$P_{bt}(\mathbf{D}=\mathbf{d}|\mathbf{X})$$

$$P_{bt}(\mathbf{Z}=\mathbf{z}|\mathbf{X})$$

$${\mathbf{d}}^{(1)},\dots ,{\mathbf{d}}^{(M)}$$

$${\mathbf{z}}^{(1)},\dots ,{\mathbf{z}}^{(M)}$$

Full text

1 Introduction

While instrumental variable (IV) methods have a long history in economics, they have become increasingly popular in epidemiology and health services research. While IV methods can be used to analyze randomized trials with noncompliance [1, 2], investigators commonly seek to exploit natural experiments by identifying instruments that act as haphazard nudges that induce exposure to a treatment. Instruments are valuable because, subject to a set of assumptions, they provide valid causal inferences in the presence of unobserved confounders [3, 4]. However, some of these assumptions cannot be tested with observed data, and recent research has focused on developing guidelines, diagnostics, and falsification tests for IV analyses.[5, 6, 7, 8, 9, 10, 11]

One strand of this literature has focused on developing methods for assessing whether the assumption of no unmeasured confounding between the IV and the outcome is plausible. This assumption requires careful assessment, because IVs based on natural experiments are not assigned using actual random assignment. Instead, investigators seek to find natural circumstances that might produce as-if random assignment of the IV. If the IV behaves in an as-if random fashion, then baseline covariates should not differ significantly across levels of the IV. Consistent with this logic, analysts often compute the observed covariate balance across levels of the IV and compare it with balance across levels of the treatment [12, 4]. If covariate balance across levels of the IV is not at least better than covariate balance across the treatment, this is one indication that the no unmeasured confounding assumption is untenable. Alternatively, bias in covariates from an IV can be compared to bias from an intention-to-treat analysis [4]. Jackson and Swanson[13] outlined a method of graphical display for easy bias comparisons, and Davies et al.[14, 15] suggest using formal statistical tests to compare the bias associated with the IV to the bias associated with the treatment. More recently, Zhao and Small [16] developed plots for bias that account for whether observed confounders are related to the outcome.

There are, however, key limitations to extant methods. First, current tests only provide covariate-by-covariate results, often making it hard to discern an overall pattern in whether the data favor the IV relative to the treatment. Second, all of the above methods rely on parametric assumptions. Third, while these tests compare the IV to the treatment, they do not compare balance on the IV and the treatment to an overall standard. As an alternative, we develop a new diagnostic test for assessing IV–outcome confounding. Our test compares IV and treatment balance or bias to a common benchmark: the balance or bias we would expect if either the IV or treatment had been randomized. The results from our test can be plotted for each covariate, but our method also allows for a global test. Moreover, our test is nonparametric, and thus relies on minimal assumptions.

In Section 2, we outline notation and the necessary assumptions for an IV analysis. In Section 3, we propose our falsification test for IV–outcome confounding, and in Section 4 we apply our diagnostics to a recent IV application that uses bed availability in the intensive care unit (ICU) as an instrument for admission to the ICU. In Section 5, we conclude.

2 Notation and IV assumptions

Let $D$ be the binary treatment exposure, $Y$ be the outcome, $Z$ be a binary instrument, and $\bm{X}=(X_{1},\ldots,X_{K})$ be a set of covariates measured before $Z$ and $D$ are assigned. For $z=0,1$ and $d=0,1$, let $Y{(z,d)}$ be the potential outcome of $Y$ when an intervention sets $Z=z$ and $D=d$, and let $D{(z)}$ be the potential outcome of $D$ when $Z=z$. We also assume there are $N$ units, indexed by $i=1,\dots,N$, though we do not subscript by $i$ in this section. The observed values $(Y,D,Z)$ are related to the potential outcomes through the following equations:

[TABLE]

[TABLE]

This notation implicitly assumes the stable unit treatment value assumption (SUTVA) [17]. SUTVA implies that 1) the levels of $D$ (1 and 0) adequately represent all versions of the treatment, and 2) each subject’s outcomes are not affected by other subjects’ exposures.

In an IV analysis, investigators assume three “core assumptions” hold[4]:

(1) Relevance:

$\mathbb{E}[D|\bm{X},Z=1]\neq\mathbb{E}[D|\bm{X},Z=0]$.

(2) Effective random assignment:

$Z$ is independent of $D(z)$ and $Y(z,d)$ given $\bm{X}$ for all $z=0,1$ and $d=0,1$. This implies that there are no unmeasured confounders for $Y$ and $Z$ conditional on $\bm{X}$.

(3) Exclusion restriction:

$Y(z,d)=Y(z^{\prime},d)$ for any $z$, $z^{\prime}$, and $d$. This means that the potential outcomes only depend on the instrument $Z$ through its effect on the exposure $D$. This assumption allows us to rewrite the potential outcomes as $Y(0,d)=Y(1,d)=Y(d)$.

The methods we develop in Section 3 are designed to shed light on the plausibility of Assumption (2). In randomized experiments with noncompliance, Assumption (2) holds by design. However, for IVs based on natural experiments, investigators must assume that the IV is as-if randomly assigned to the units under study. Assumption (2) cannot be directly tested, but observed data can be used to falsify it. That is, an IV that is as-if randomly assigned should be independent of observed covariates. If we find that an IV is independent of observables, that tells us little about whether an IV is independent of unobservables; however, if an IV is strongly associated with observables, that suggests that Assumption (2) may not hold. Next, we outline a test to probe Assumption (2).

3 A Randomization Test for Instrument Validity

As discussed in Section 1, previous works have tested for effective random assignment by comparing the IV and the exposure either in terms of covariate balance [12] or bias [4, 13, 14, 15, 16]. Covariate balance—also known as the covariate prevalence difference—is defined as the difference in covariate means across a binary measure of the IV. Meanwhile, in this context, bias is defined as the covariate prevalence difference divided by the treatment prevalence difference for the same covariate. Our test can incorporate covariate balance or bias. The intuition behind our test is that it compares the observed covariate balance or bias to the balance or bias we would expect from an as-if randomized experiment. Thus, our test proceeds by first establishing a standard for as-if randomization.

If Assumption (2) holds, we can write the conditional distribution of the instrument as $P(\bm{Z}|\bm{D},\bm{Y}(\bm{D}),\bm{X})=P(\bm{Z}|\bm{X})$, which is the conditional distribution of the instrument as a function of observed covariates. Thus, an instrument is as-if randomized if its distribution follows some $P(\bm{Z}|\bm{X})$, and our test will assess if indeed the distribution of the instrument follows some distribution $P(\bm{Z}|\bm{X})$. For example, in our application in Section 4, we will consider positing that the instrument follows complete randomization, defined as:

[TABLE]

Under complete randomization, the instrument is uniformly randomized to all units conditional on the number of treated units $N_{T}$. In practice, the investigator can posit any assignment mechanism that can be written as a distribution $P(\bm{Z}|\bm{X})$, and our test can in turn assess the plausibility of that assignment mechanism. For example, if the units can be easily organized into blocks, then block randomization (instead of complete randomization) may be appropriate. See Imbens and Rubin[18] (Chapter 4) for a full taxonomy of assignment mechanisms.

We will now outline a randomization test for the hypothesis $H_{0}:\bm{Z}\sim P(\bm{Z}|\bm{X})$, i.e., the hypothesis that the instrument is as-if randomized according to $P(\bm{Z}|\bm{X})$. Our test compares the observed covariate balance to the distribution of covariate balance we would expect if indeed $H_{0}$ were true. If the observed covariate balance is substantially outside this distribution, then this is evidence against $H_{0}$—i.e., this is evidence against the assumption that the instrument is as-if randomized. Alternatively, if the observed covariate balance falls substantially within this distribution, this is evidence that the instrument follows the distribution $P(\bm{Z}|\bm{X})$ specified by the investigator. The formal algorithm for our randomization test is:

Algorithm for $\alpha$-level Randomization Test for $H_{0}:\bm{Z}\sim P(\bm{Z}|\bm{X})$

Specify an assignment mechanism $P(\bm{Z}|\bm{X})$.

Define a test statistic $t(\bm{Z},\bm{X})$, which measures covariate balance or bias.

Generate $M$ random draws $\bm{Z}^{(1)},\dots,\bm{Z}^{(M)}\sim P(\bm{Z}|\bm{X})$. Here, $M$ should be sufficiently large (e.g., 5,000 or more) such that the randomization distribution is well-approximated.

Compute the following randomization-based $p$-value:

\displaystyle p=\frac{1+\sum_{m=1}^{M}\mathbb{I}(|t(\bm{z}^{(m)},\bm{X})|\geq|t^{obs}|)}{M+1},\hskip 7.22743pt\text{where t^{obs}\equiv t\left(\bm{Z}^{obs},\bm{X}\right)}

(2)

Reject $H_{0}$ if $p\leq\alpha$.

A key advantage of our randomization test is that it is valid—i.e., the test controls the false rejection rate—and exact for the hypothesis that $H_{0}:\bm{Z}\sim P(\bm{Z}|\bm{X})$ for any assignment mechanism of interest [19]. Moreover, the test does not rely on any parametric assumptions.

The procedure requires the investigator to select a test statistic $t(\bm{Z},\bm{X})$, which can be any measure of covariate balance or bias. For example, the test statistic can be covariate-specific. Two natural covariate-specific test statistics consistent with the literature are the covariate prevalence difference—which we also refer to as balance—or bias:

[TABLE]

where $\overline{\bm{X}}_{Z=z}$ is the vector of covariate means for units with $Z=z$, and $\overline{\mathbf{D}}_{Z=z}$ is analogously defined. Covariate-specific test statistics allow the analyst to assess to what degree the instrument is as-if randomized for each covariate.

Alternatively, one can assess if the instrument is as-if randomized across a global measure of covariate balance, which is perhaps the biggest advantage of our test. One measure of global covariate balance is the Mahalanobis distance [20], defined as

[TABLE]

Here we have defined the Mahalanobis distance in terms of balance. Holding the instrument strength $\overline{\bm{D}}_{Z=1}-\overline{\bm{D}}_{Z=0}$ fixed, this is equivalent to defining this quantity in terms of bias, because the Mahalanobis distance is affinely invariant, which implies:

[TABLE]

Thus, a benefit of the Mahalanobis distance is that it acts as an omnibus analog to covariate balance as well as bias, while also standardizing by the covariance among the covariates instead of their marginal variances.

Regardless of the choice of test statistic, our test is always valid for the assumption that the instrument is as-if randomized. However, the choice of test statistic may also affect the power of our test, as is the case for any falsification test. We elaborate on this point in Section 5.

Our test is identical to the randomization test proposed by Branson[19]. However, Branson uses the above test to assess as-if randomization for exposure assignment in matched datasets, whereas we assess as-if randomization for instrument assignment in observational datasets. The original use of the test, however, suggests another extension. In our above formulation, we use the test to assess if the instrument is as-if randomized, and we can do the same for the exposure by replacing $\bm{Z}$ with $\bm{D}$ in our above formulation. Then, we can compare the conclusions of our test for $\bm{Z}$ and for $\bm{D}$, as we outline further in the next section.

3.1 Comparing Instrument and Exposure Balance or Bias

Thus far, we have outlined a randomization test that can assess if the instrument or the exposure are as-if randomized to the units under study. Table 1 outlines the four possible scenarios that can occur when making these assessments. Case 1 provides clear evidence in favor of using the IV design, because the IV appears to be as-if randomized but the exposure does not. Case 2 is clear evidence against the IV design, because the exposure appears to be as-if randomized but the IV does not. Case 3 is an ambiguous but unlikely situation, where both the IV and the exposure appear to be as-if randomized. Case 4 is probably the most likely scenario in an observational study, where neither the IV nor the exposure appear to be as-if randomized. One dogmatic conclusion that one could draw from Case 4 is that no analysis should be conducted. A more pragmatic view is that, even if neither the instrument nor the exposure appear to be as-if randomized, it may still be desirable to conduct an IV analysis if the instrument appears to at least be significantly closer to as-if random than the exposure. This pragmatic view is consistent with the notion that natural experiments are useful if they at least reduce bias relative to cases where treatments are self-selected.[21, 22, 23]

Accordingly, we now outline how to test whether the instrument is significantly closer to being as-if randomized than the exposure. For this test, we must consider alternative assignment mechanisms that the exposure or instrument may follow, because, under Case 4 in Table 1, we have rejected that they are completely randomized to the units under study. One mechanism to posit is that exposure and instrument assignment follow Bernoulli trials:

[TABLE]

where $e_{D}(\bm{x}_{i})\equiv P(\bm{D}_{i}=1|\bm{x}_{i})$ is the $i$th propensity score for receiving the exposure, and $e_{Z}(\bm{x}_{i})$ is the $i$th propensity score analogously defined for the instrument. This is a natural mechanism to posit in observational studies, because such a mechanism is implicitly assumed when conducting propensity score analyses, which are widely used for observational studies[24, 25, 26, 27]. These propensity scores are not known, but they can be easily estimated via logistic regression.

After the propensity scores are estimated, we can generate many hypothetical randomizations according to $P_{bt}(\bm{D}|\bm{X})$ and $P_{bt}(\bm{Z}|\bm{X})$ via biased coin flips, and then compute the balance or bias across these hypothetical randomizations; this will measure the balance or bias we would expect from the instrument assignment mechanism and the exposure assignment mechanism. This can be done by the following procedure:

Estimate the exposure propensity scores $e_{D}(\bm{x})$ and the instrument propensity scores $e_{Z}(\bm{x})$. 2. 2.

Generate many hypothetical exposure assignments and instrument assignments:

[TABLE]

where $\hat{P}_{bt}(\bm{D}|\bm{X})$ is $P_{bt}(\bm{D}|\bm{X})$ in (5), but using the estimated $\hat{e}_{D}(\bm{x})$ from Step 1; $\hat{P}_{bt}(\bm{Z}|\bm{X})$ is analogously defined. 3. 3.

Compute the test statistic (balance or bias) $t(\bm{D},\bm{X})$ and $t(\bm{Z},\bm{X})$ across the hypothetical exposure assignments and instrument assignments, respectively, from Step 2. 4. 4.

Compare the randomization distributions $(t(\bm{d}^{(1)},\bm{X}),\dots,t(\bm{d}^{(M)},\bm{X}))$ and $(t(\bm{z}^{(1)},\bm{X}),\dots,t(\bm{z}^{(M)},\bm{X}))$.

If the randomization distributions are non-overlapping, this implies that balance or bias across the exposure and across the instrument significantly differ. Furthermore, both of these randomization distributions can be compared to the corresponding randomization distribution under complete randomization defined in (1). For this comparison, global measures of balance or bias—such as the Mahalanobis distance—are particularly useful, because they place gold standard randomized assignment, exposure assignment, and instrument assignment on a univariate scale, making comparisons straightforward. As we will see in the next section, it is especially helpful to plot the randomization distribution across these three assignment mechanisms, because this aids in assessing if the instrument is significantly closer to being as-if randomized than the exposure. Next, we demonstrate the proposed methods in a case study.

4 Application: Is Bed Availability a Valid Instrument for ICU Admission?

We use data from (SPOT)light, a prospective cohort study of deteriorating ward patients referred to critical care in 48 National Health Service (NHS) hospitals between 1 November 2010 and 31 December 2011 [28]. The cohort includes 13,011 patients. The primary clinical question of interest was whether transfer to the ICU reduced in-hospital mortality. Bias from confounding by indication may be a threat if the reasons for transfer to the ICU are not fully recorded in the data. Threat of this bias motivated the use of an IV analysis. During data collection, ICU bed availability at the time of assessment for ICU admission was measured for use as an instrument for ICU transfer [28, 29, 30, 31]. That is, if there are few ICU beds available when the patient is assessed, this discourages ICU transfer. In the data, ICU bed availability ranges from 0 to 19 with a median of 4 and an IQR of 4. Following an earlier analysis, we use a binary measure of the instrument where 1 indicates that fewer than 4 ICU beds were available [29].

The exposure is defined as transfer to and receipt of care in the ICU. Measured confounders include age, gender, septic diagnosis (0/1), and peri-arrest (0/1). Physiological data were also collected to create three measures of patient risk severity. The data also record patients’ existing level of care at assessment and the recommended level of care after assessment, defined using the UK Critical Care Minimum Dataset (CCMDS) levels of care. These levels are 0 and 1 for normal ward care, 2 for care within a high-dependency unit, and 3 for care within the ICU.

Figure 1(a) shows the distribution of the propensity scores (estimated using logistic regression) across the exposure and across the instrument. The IV propensity scores are much more closely concentrated around moderate values, indicating better balance. Meanwhile, Figure 1(b) shows the standardized covariate mean differences (SCMDs)—in other words, the standardized prevalence difference defined in (3)—across the instrument and across the exposure. SCMDs across the instrument are more closely concentrated around zero—and many are below the 0.1 rule-of-thumb that is commonly employed in the literature [32, 33, 34, 35]—again indicating better balance for the instrument. Figures 1(a) and 1(b) are useful diagnostics and are similar in spirit to current practice, but they do not tell us if the instrument balance is better than exposure balance to a statistically significant degree, or if the instrument can be assumed to be as-if randomized.

Next, we apply the randomization test from Section 3. First, we implement the test for each covariate using the SCMDs as the test statistic. Specifically, we implemented the randomization test by permuting the instrument indicator 10,000 times and computing the SCMDs for each permutation. The distribution of the resulting 10,000 SCMDs represent the balance we would expect from a completely randomized experiment. If the observed balance is significantly outside this distribution, then this is evidence against as-if randomization.

For example, Figure 2 shows the permutation distribution of the SCMD for the age covariate. The observed SCMD for the instrument is within the 2.5% and 97.5% quantiles of this permutation distribution, meaning that our test concludes that the instrument is as-if randomized for this covariate with 95% confidence (the randomization test $p$-value is 0.15). Furthermore, the observed SCMD for the exposure is not within these quantiles, indicating that the exposure is not as-if randomized for this covariate. Meanwhile, Figure 3(a) shows the SCMDs for the other covariates and their corresponding permutation quantiles. All of the observed SCMDs across the exposure are outside these quantiles, and thus as-if randomization is not tenable for the exposure. However, the observed SCMDs across the instrument are within these quantiles for some of the covariates but not others, making it unclear whether as-if randomization can be assumed for the instrument.

The situation is even more ambiguous if we look at bias instead of balance, which has been recently recommended in the literature[4, 13, 14, 15, 16]. Figure 3(b) shows the instrument and exposure bias for each covariate—as defined in (3)—and their corresponding permutation quantiles. Note that exposure bias is equal to exposure balance, while instrument bias is instrument balance inflated by the strength of the IV. Thus, the red dots in Figures 3(a) and 3(b) representing exposure balance and bias are identical, while the blue triangles representing instrument bias in Figure 3(b) are inflated versions of those in Figure 3(a). Now it is less clear if the IV is preferable over the exposure in terms of bias.

The ambiguities in covariate-by-covariate diagnostics make global measures of balance or bias useful. Next, we implement our test using the square-root of the Mahalanobis distance—i.e., the square-root of (4)—which acts as a global measure of balance as well as bias. We again permute the instrument indicator 10,000 times and compute the square-root of the Mahalanobis distance for each permutation. The resulting permutation distribution is shown in black in Figure 4. The observed square-root of the Mahalanobis distance for the instrument ($\sqrt{\text{MD}_{Z}^{\text{obs}}}$) and exposure ($\sqrt{\text{MD}_{D}^{\text{obs}}}$)—marked by vertical lines—are both outside this distribution. Therefore, we conclude that neither the instrument nor the exposure are completely randomized.

Because we have rejected the null hypothesis that instrument and exposure assignment follow complete randomization, we must posit alternative assignment mechanisms that the instrument and exposure may follow. Next, we assume that the instrument and exposure assignment mechanisms follow Bernoulli trials, denoted by (5) and (6), and we will estimate the propensity scores for these assignment mechanisms using logistic regression, as we did in Figure 1(a). Then, we generate 10,000 random draws from these assignment mechanisms and compute the square-root of the Mahalanobis distance for each draw. The resulting 10,000 Mahalanobis distances represent the balance or bias we would expect from a Bernoulli trial experiment using the estimated propensity scores. The randomization distributions under instrument assignment and exposure assignment are shown in blue and red, respectively, in Figure 4. Two observations can be made: (1) these distributions are non-overlapping, and (2) the instrument distribution is substantially closer to the as-if randomization distribution than the exposure distribution. Thus, while we cannot conclude that the instrument is as-if randomized, we can at least be confident that the IV is substantially closer to being as-if randomized than the exposure. This observation can be used to justify conducting an IV analysis on these data, which has been done in several recent works.[28, 29, 30, 31]

5 Conclusions

Much recent work has developed statistical tools to help investigators judge whether a posited instrument is valid. Here, we add to this literature by developing a randomization test to probe the assumption that the IV is effectively randomly assigned. Our test compares instrument and exposure balance or bias to the balance or bias we would expect from a randomized experiment. This allows us to assess if the instrument is as-if randomized as well as assess if the instrument is significantly closer to being as-if randomized than the exposure. This test has several key advantages. First, our randomization test is an exact and valid test for the effective random assignment assumption, and it does not rely on any parametric assumptions. Second, our test can provide covariate-by-covariate results as well as a global test, and it can incorporate measures of balance as well as bias. Third, our test uses what we would expect from a randomized experiment as a clear benchmark. Finally, the results from the test can be summarized in a single graphical display and are easily interpretable. We demonstrated our approach with a recent application that uses bed availability in the ICU as an instrument for admission to the ICU. In that application, we found that neither the instrument nor the exposure is effectively randomized; however, we also found that the instrument is substantially closer to being as-if randomized than the exposure.

There are several limitations to our test, and addressing these limitations would be an interesting avenue for future work. First, we have focused on the case where the exposure and the instrument are binary, and it would be helpful to extend our test to cases where the exposure and/or the instrument are multi-valued. Furthermore, while our test provides a valid way to assess as-if randomization for an instrument in observational studies, the test may be under-powered. This is not a limitation of our randomization test specifically but of falsification tests in general. For example, most tests in the literature focus on comparing instrument and exposure balance in terms of the covariates’ first moments—and we have done the same here—but these tests will have little power in detecting differences in other functions of the covariates. Of course, our test allows for nonlinear functions of the covariates, but these functions must be specified by the investigator. Future work could consider methods that are both valid and have optimal power in detecting violations of the effective random assignment assumption.