Limitations of design-based causal inference and A/B testing under arbitrary and network interference

TL;DR

This paper demonstrates fundamental limitations in design-based causal inference and A/B testing when interference between units is arbitrary or network-based, showing that large samples do not guarantee low variance or unbiased estimates.

Contribution

It formalizes the limitations of nonparametric and parametric inference under interference, revealing that assumptions are necessary for reliable causal estimation in network experiments.

Findings

Unbiased estimators cannot guarantee decreasing variance under interference.

Large sample sizes do not ensure small variance in experiments with interference.

Traditional causal inference assumptions do not extend straightforwardly to network interference settings.

Abstract

Randomized experiments on a network often involve interference between connected units; i.e., a situation in which an individual's treatment can affect the response of another individual. Current approaches to deal with interference, in theory and in practice, often make restrictive assumptions on its structure---for instance, assuming that interference is local---even when using otherwise nonparametric inference strategies. This reliance on explicit restrictions on the interference mechanism suggests a shared intuition that inference is impossible without any assumptions on the interference structure. In this paper, we begin by formalizing this intuition in the context of a classical nonparametric approach to inference, referred to as design-based inference of causal effects. Next, we show how, always in the context of design-based inference, even parametric structural assumptions that allow the existence of unbiased estimators, cannot guarantee a decreasing variance even in the large sample limit. This lack of concentration in large samples is often observed empirically, in randomized experiments in which interference of some form is expected to be present. This result has direct consequences for the design and analysis of large experiments---for instance, in online social platforms---where the belief is that large sample sizes automatically guarantee small variance. More broadly, our results suggest that although strategies for causal inference in the presence of interference borrow their formalism and main concepts from the traditional causal inference literature, much of the intuition from the no-interference case do not easily transfer to the interference setting.

Tables

Table 1. Table 1: Expected number of effective treatments and fraction of informative assignments for different interference structures, under the Bernoulli Design

interference	no	1-local	arbitrary
$E_i$	2	$2^{\left|{𝒩}_i\right|}$	$2^N$
$F_i\left(\text{Z}\right)$	$\frac{1}{2}$	$\frac{1}{2^{\left|{𝒩}_i\right|}}$	$\frac{1}{2^N}$

Table 2. Table 2: Asymptotic expected number of effective treatments and fraction of Z − e ​ x ​ p ​ o ​ s ​ e ​ d Z 𝑒 𝑥 𝑝 𝑜 𝑠 𝑒 𝑑 \textbf{Z}-exposed sets for the two different Erdos-Renyi specifications of Example  3 , under 1-local interference.

	p = 1/N	$p=1/\sqrt{N}$
${𝔼}_{𝒢}\left[E_i\right]$	2e	$\infty$
${𝔼}_{𝒢}\left[F_i\left(\text{Z}\right)\right]$	$\frac{1}{2e^{1/2}}$	0

Equations

$$0<Y_i(\text{Z})<M\forall i=1,\dots N$$

$$\theta (\text{A},\text{B})=\overline{Y}(\text{A})-\overline{Y}(\text{B})$$

$${\mathbb{E}}_{\mathcal{R}}[\hat{\theta }(\text{Z})]=\theta (\text{A},\text{B})$$

$$Y_i(\text{Z})=Y_i(Z_i)\forall i=1,\dots ,N$$

$$\theta (\text{A},\text{B})=\frac{1}{N}\sum _i^N(Y_i(\text{A})-Y_i(\text{B}))$$

$$\hat{\theta }(\text{Z})=\frac{1}{N_A}\sum _i^{N}I(Z_i=A)Y_i(\text{Z})-\frac{1}{N_B}\sum _i^{N}I(Z_i=B)Y_i(\text{Z})$$

$$\hat{\theta }(\text{Z})=\frac{1}{N_A}\sum _i^{N}I(Z_i=A)Y_i(A_i)-\frac{1}{N_B}\sum _i^{N}I(Z_i=B)Y_i(B_i)$$

$${\mathbb{E}}_{\mathcal{R}}[\hat{\theta }(\text{Z})]=\theta (\text{A},\text{B})$$

$${\mathbb{V}}_{\mathcal{R}}[\hat{\theta }]$$

$$\le \frac{1}{N_A}\frac{1}{N-1}\sum _i(Y_i(\text{A})-\overline{Y}(\text{A}){)}^2$$

$$+\frac{1}{N_B}\frac{1}{N-1}\sum _i(Y_i(\text{B})-\overline{Y}(\text{B}){)}^2$$

$$\le \frac{1}{N_A(N-1)}N{M}^2+\frac{1}{N_B(N-1)}N{M}^2$$

$$\le \frac{4M^2}{N-1}$$

$$\hat{\theta }(\text{Z})=C(\text{Z})+2^{N}I(\text{Z}=\text{A}){\theta }_1(\text{A})+2^{N}I(\text{Z}=\text{B}){\theta }_2(\text{B})$$

$$\sum _{\text{Z}\in \mathcal{Z}}C(\text{Z})=0$$

$$\theta (\text{A},\text{B})=\overline{Y}(\text{A})-\overline{Y}(\text{B})$$

$$\hat{\theta }(\text{Z})=\{\begin{array}{ll}\overline{Y}(\text{A})& \text{mbox}if\text{Z}=\text{A}\\ -\overline{Y}(\text{B})& \text{mbox}if\text{Z}=\text{B}\\ 0& \text{mbox}otherwise\end{array}$$

$$MSE(\hat{\theta },\theta )\ge \frac{M^2}{8}$$

$$Y_i(\text{Z})=Y_i({\text{Z}}_{{\mathcal{N}}_i^{(k)}})$$

$${\mathcal{Z}}_i^{(k)}(\text{A})=\{\text{Z}:{\text{Z}}_{{\mathcal{N}}_i^{(k)}}={\text{A}}_{{\mathcal{N}}_i^{(k)}}\}$$

$$\hat{\theta }(\text{Z})$$

$$=\frac{1}{N}\sum _i\frac{I(Z\in {\mathcal{Z}}_i^{(k)}(\text{A}))}{P(\text{Z}\in {\mathcal{Z}}_i^{(k)}(\text{A})}{Y}_i(\text{A})$$

$$-\frac{1}{N}\sum _i\frac{I(Z\in {\mathcal{Z}}_i^{(k)}(\text{B}))}{P(\text{Z}\in {\mathcal{Z}}_i^{(k)}(\text{B})}{Y}_i(\text{B})$$

$${\mathbb{V}}_{\mathcal{R}}[\hat{\theta }]={\mathbb{V}}_{\mathcal{R}}[{\hat{\theta }}_{\text{A}}]+{\mathbb{V}}_{\mathcal{R}}[{\hat{\theta }}_{\text{B}}]-2\mathbb{C}o{v}_{\mathcal{R}}[{\hat{\theta }}_{\text{A}},{\hat{\theta }}_{\text{B}}]$$

$${\mathbb{V}}_{\mathcal{R}}[{\hat{\theta }}_{\text{A}}]$$

$$\displaystyle+\sum_{i}\sum_{j\neq i}(2^{|\mathcal{N}_{i}^{(k)}(\textbf{A})\cap\mathcal{N}_{j}^{(k)}(\textbf{A})|}-1)Y_{i}(\textbf{A})Y_{j}(\textbf{A})\bigg{]}$$

$$\mathbb{C}o{v}_{\mathcal{R}}[{\hat{\theta }}_{\text{A}},{\hat{\theta }}_{\text{B}}]$$

$$\displaystyle+\sum_{i}\sum_{j\neq i}Y_{i}(\textbf{A})Y_{j}(\textbf{B})I(|\mathcal{N}^{(k)}_{i}\cap\mathcal{N}^{(k)}_{j}|>0)\bigg{]}$$

$$0<K<\text{Y}(\text{Z})<M$$

$$\displaystyle\mathbb{E}_{\mathcal{G}}\bigg{[}\mathbb{V}_{\mathcal{R}}[\hat{\theta}]\bigg{]}$$

$$+(1+3p)(1+p^2{)}^{(N-2)}-1$$

$$\displaystyle+\frac{1}{N}+\frac{N(N-1)}{2}(1-(1-p)(1-p^{2})^{N-2})\bigg{)}$$

$$\theta =\frac{1}{N}\sum _i{Y}_i(Z_i=A,{\text{Z}}_{-i}=\text{B})$$

$$\hat{\theta }(\text{Z})=C(\text{Z})+\frac{2^N}{N}\sum _{i}I(\text{Z}={\text{Z}}^{(i)})Y_i({\text{Z}}^{(i)})$$

$$Y_i(\text{Z})=Y({\text{Z}}_{G_i})\forall \text{Z}$$

Full text

1. 1 Introduction

2. 1.1 Background and set-up

3. 1.2 Contributions

4. 2 No-interference and arbitrary interference

5. 2.1 Setup

6. 2.2 Inference under no-interference

7. 2.3 Arbitrary interference

8. 3 Caution when structuring the interference mechanism

9. 4 Discussion

10. 4.1 Broader class of estimands

11. 4.2 Effective treatments and informative sets

12. 4.3 Understanding interference structures

13. 4.4 Connection with the Incidental Parameter Problem

14. A Technical appendix

15. A.1 Variance terms in Section 2

16. A.2 Proof of theorems

17. A.3 Derivations for Example 3

18. A.4 Derivations for Section 4

1 Introduction

In modern randomized experiments, “interference” typically means that the response of a given unit to a certain treatment may depend on the treatment assigned to other units. In online marketing, (Aral and Walker, 2012) showed that adoption of a product by an individual in a social network tends to encourage the adoption of the same product by their neighbors. Similar examples can be found in epidemiology, where vaccinating a certain percentage of the population is expected to lower the health risk for the entire population (Hudgens and Halloran, 2012), or in political science where encouraging an individual to vote is expected to increase the turnout for other members of the household (Sinclair et al., 2012). Researchers have had some success in addressing the question by making assumptions on the interference mechanism (Hudgens and Halloran, 2012; Sinclair et al., 2012; Basse and Feller, 2016; Ugander et al., 2013; Bowers et al., 2016b, a). However, because these mechanisms are often very complex, it is tempting to believe that we can rely on large sample sizes to avoid making assumptions. The goal of this paper is to show how wrong this intuition is, precisely.

1.1 Background and set-up

In a randomized experiment (or A/B test), N units are randomly assigned to treatments A or B, and an outcome of interest $Y_{i}(\textbf{Z}^{obs})$ is measured for each unit $i$, where $\textbf{Z}^{obs}\in\mathcal{Z}=\{A,B\}^{N}$ is the observed assignment vector. There are generally two paradigms available for making inference: the model-based approach, and the design-based approach.

In the model-based approach, the vector of observed outcomes is modeled $Y_{i}(\textbf{Z}^{obs})\sim F(\vec{\xi})$. The estimands are generally functions of the parameters $\theta=g(\vec{\xi})$ and the estimators are usually obtained using maximum likelihood or bayesian inference. Two key points are that the estimators are selected using the model $F(\vec{\xi})$, and the properties of the estimator are typically derived in some asymptotic regime and incorporate randomness due to both the model for the outcomes and the randomization distribution.

In contrast, the design-based approach we consider in this paper takes the opposite perspective and considers the potential outcomes $\{\textbf{Y}(\textbf{Z})\}_{\textbf{Z}}$ as fixed but a-priori unknown quantities. The assignment mechanism $\mathcal{R}$, which assigns a probability $\mathbb{P}_{\mathcal{R}}(\textbf{Z})$ to every vector $\textbf{Z}\in\mathcal{Z}$ provides the only source of randomness. Estimands are then generally functions of the potential outcomes $\theta=g(\{\textbf{Y}(\textbf{Z})\})$, and it is desirable to find a pair $(\hat{\theta},\mathbb{P}_{\mathcal{R}})$ such that the estimator $\hat{\theta}$ has good properties under the design $\mathcal{R}$ in finite samples. One goal of this paper is to show that under arbitrary interference, there exist no pair $(\hat{\theta},\mathbb{P}_{\mathcal{R}})$ with good properties.

A third approach to inference exists, typically referred to as model-assisted, which a attempts to blend the two approaches by using a model for the outcomes to inform the choice of design or correction factors for estimators in a given family (Särndal et al., 2003; Basse and Airoldi, 2015). Since the actual inference is then performed from a design-based perspective, this approach also falls under the scope of our paper.

The idea that inference is not feasible without assumptions on the interference mechanism has been suggested in the literature for observational studies (e.g., see Shalizi and Thomas, 2011), and also in the context of randomized experiments; for instance Aronow and Samii (2013) state that under arbitrary interference, “it is clear that there would be no meaningful way to use the results of the experiment”, and then focuses on how to do inference under some form of restricted interference. In the first, and what is to our knowledge also the only, attempt at formalizing the issue, Manski (2013) shows that in a model-based setting, the distribution of potential outcomes is not identifiable under arbitrary interference.

1.2 Contributions

Our paper makes three contributions. First, we extend and clarify the results of (Manski, 2013) in the context of design-based inference (Section 2) and prove, among other things, that there exist no consistent estimators under arbitrary interference. Second, we extend the work of (Aronow and Samii, 2013) by focusing on a popular design and an interference structure commonly assumed in network settings. Assuming an Erdos-Renyi for the network, we show that for a class of unbiased estimators, consistency depends on the parameter $p$ of the model for the graph. Finally, using the concept of effective treatment introduced by (Manski, 2013), we provide analytical insights into the general problem of interference and the convergence of estimators.

For researchers, this paper is meant to provide a clear formalization of some intuitions that have been suggested. For practitioners, this paper is meant to offer a convincing argument for the necessity of making explicit assumptions about the structure of the interference mechanism, to rely be able to rely on asymptotic standard errors for causal inference.

2 No-interference and arbitrary interference

2.1 Setup

To avoid pathological cases, we assume that the potential outcomes are bounded:

Assumption 1** (bounded outcomes).**

There exists $M$ such that for all $N$:

[TABLE]

Although many of our results could generalize to large classes of designs $\mathcal{R}$, we will focus on three assignment mechanisms for the sake of clarity. The Completely Randomized Design (CRD) assigns a fixed number $N_{A}$ of randomly selected units to treatment A, and the other $N_{B}=N-N_{A}$ units to treatment B. The Bernoulli Design (BD) assigns independently each unit to treatments A or B with probability 1/2, while the Conditional Bernoulli Design (CBD) operates similarly to the Bernoulli Design , but excludes the assignments $\textbf{Z}=\textbf{A}$ and $\textbf{Z}=\textbf{B}$ in which all units are assigned to A, or all units assigned to B.

The estimands considered in this paper are of the form $\theta(\textbf{A},\textbf{B})=g(\textbf{Y}(\textbf{A}),\textbf{Y}(\textbf{B}))$, and estimators will be denoted by $\hat{\theta}(\textbf{Z})$.

Example 1**.**

The most popular estimand in causal inference is the average total treatment effect (Imbens and Rubin, 2015):

[TABLE]

but our results are stated in greater generality.

All statements about estimators must hold regardless of the value of the potential outcomes $\{\textbf{Y}(\textbf{Z})\}_{\textbf{Z}}$. In particular we consider the following definition of unbiasedness:

Definition 1**.**

Let $\theta(\textbf{A},\textbf{B})=g(\textbf{Y}(\textbf{A}),\textbf{Y}(\textbf{B}))$ be the estimand of interest. An estimator $\hat{\theta}(Z)$ is said to be unbiased if:

[TABLE]

for all values of the potential outcomes $\{\textbf{Y}(\textbf{Z})\}_{\textbf{Z}}$ (the expectation being taken with respect to the assignment mechanisms).

In particular, an estimator is not unbiased if the equality $E(\hat{\theta}(\textbf{Z}))=\theta(\textbf{A},\textbf{B})$ only holds for specific values of $\{\textbf{Y}(\textbf{Z})\}_{Z}$. Similar considerations apply to the concept of consistency.

2.2 Inference under no-interference

The most prominent applications involve situations in which interference is a nuisance and can be safely be assumed away. The main clause of the popular Stable Unit Treatment Value Assumption (SUTVA; Rubin (1980)) implies that there is no interference between units:

Assumption 2** (No interference).**

[TABLE]

Studying the no-interference case is important because it often implicitly guides our intuition in more complex scenarios, especially the large sample behavior or estimators. One aim of that paper is to show the dangers of that intuition.

We illustrate the standard characteristics of inference under this assumption by focusing on the Completely Randomized Design and the average total treatment effect:

[TABLE]

as our estimand. The following estimator:

[TABLE]

which under Assumption 2 simplifies to:

[TABLE]

can easily be shown to be unbiased for $\theta(\textbf{A},\textbf{B})$ under the Completely Randomized Design . That is:

[TABLE]

and the variance is (see e.g Imbens and Rubin (2015) chapter 6):

[TABLE]

where the terms $V_{A}$, $V_{B}$, and $V_{\theta}$ are defined in appendix. So under Assumption 1 and Assumption 2 there exists an unbiased estimator, with variance of order $O(1/N)$.

2.3 Arbitrary interference

Under arbitrary interference, the outcome for unit $i$ depends on the entire assignment vector Z, not just on its own assignment $Z_{i}$. We show in this section that not only is the estimator in Equation (1) biased, but that unbiased estimators (in the sense of Definition 1) simply do not exist for a wide class of designs which includes both the Completely Randomized Design and the Conditional Bernoulli Design

Theorem 1**.**

Consider any non-degenerate 111This excludes estimands that are constant, i.e that don’t depend on $\textbf{Y}(\textbf{A})$ and $\textbf{Y}(\textbf{B})$. estimand $\theta(\textbf{A},\textbf{B})$, and any assignment mechanism $\mathcal{R}$ such that $\mathbb{P}_{\mathcal{R}}(\textbf{Z}=\textbf{A})=\mathbb{P}_{\mathcal{R}}(\textbf{Z}=\textbf{B})=0$. There exists no unbiased estimator of $\theta(\textbf{A},\textbf{B})$ under $\mathcal{R}$.

If the design assigns non zero probability to the treatment allocation vectors A and B, then unbiased estimators exist for a restricted class of estimands. The following proposition considers the case of the Bernoulli Design , as a concrete example:

Proposition 1**.**

Consider the Bernoulli Design. If the estimand is of additive form $\theta(\textbf{A},\textbf{B})=\theta_{1}(\textbf{A})+\theta_{2}(\textbf{B})$, then unbiased estimators are of the form

[TABLE]

where $C(\textbf{Z})$ does not depend on any potential outcomes 222This term may incorporate covariates, as with model-assisted estimators. In the absence of external information, we focus on the case $C(\textbf{Z})=0$. and satisfies:

[TABLE]

For other types of estimands $\theta(\textbf{A},\textbf{B})$, there exist no unbiased estimators.

Taken together, Theorem 3 and Proposition 4 formalize a simple idea: if $\textbf{Z}\neq\textbf{Z}^{\prime}$, then $\textbf{Y}(\textbf{Z}^{\prime})$ is completely non-informative for $\textbf{Y}(\textbf{Z})$. In particular, the only assignments which, if observed, could provide information about the estimand $\theta(\textbf{A},\textbf{B})$ are $\textbf{Z}=\textbf{A}$ and $\textbf{Z}=\textbf{B}$. The estimator of Proposition 4 reflects this by evaluating to 0 for any uninformative assignment (that is, $\textbf{Z}\not\in\{\textbf{A},\textbf{B}\}$), and assigning large weight to the only two informative assignments ($\textbf{Z}=\textbf{A}$ and $\textbf{Z}=\textbf{B}$). This is a known case of failure for this kind of Horvitz-Thompson estimators (Basu, 2011). The following example shows the implications of Theorem 3 and Proposition 4 when the estimand is the average total treatment effect:

Example 2**.**

Consider the following estimand:

[TABLE]

and let $0<N_{1}<N$. $\theta(\textbf{A},\textbf{B})$ has the additive form of Proposition 4 with $\theta_{1}(\textbf{A})=\overline{Y}(\textbf{A})$ and $\theta_{2}(\textbf{B})=-\overline{Y}(\textbf{B})$, so under Bernoulli Design and in the absence of external information (that is, setting $C(\textbf{Z})=0$ for all Z), the only unbiased estimator of $\theta(\textbf{A},\textbf{B})$ is:

[TABLE]

Theorem 3, however, states there exist no unbiased estimators for this estimand under Completely Randomized Design and Conditional Bernoulli Design , both of which assign zero probability to assignments A and B.

Finite sample bias is not uncommon in statistical applications, and is generally acceptable if it can be traded for a large reduction in variance and vanishes as the sample size increase. The next theorem studies this tradeoff for both the Bernoulli Design and Completely Randomized Design by looking at the Mean Squared Error (MSE):

Theorem 2**.**

Consider any estimand $\theta(\textbf{A},\textbf{B})=g(\textbf{Y}(\textbf{A}),\textbf{Y}(\textbf{B}))$, and suppose that $g$ is onto 333see appendix for technical details [0, M]. Then for all sample size $N$ and estimator $\hat{\theta}(\textbf{Z})$ there exist potential outcomes satisfying Assumption 1 such that:

[TABLE]

where the MSE is taken under Completely Randomized Design or Bernoulli Design .

In the previous section, we showed that under Assumption 2 there exists an estimator for the average total treatment effect which was unbiased, had a variance of order $O(1/N)$ and thus an MSE of order $O(1/N)$. These properties of the estimator hold for all possible values of the potential outcomes. Theorem 2 states that under arbitrary interference, this is no longer the case: whether one is conducting a small study or a large scale experiment on a social network, no estimator will perform well for all values of the potential outcomes. A direct consequence of the theorem is that there exists no consistent estimator under arbitrary interference.

Remark 1**.**

The condition on $g$ in Theorem 2 can be relaxed, without affecting the main idea behind the theorem. It is useful to note that the difference in means estimand $\theta(\textbf{A},\textbf{B})=\overline{Y}(\textbf{A})-\overline{Y}(\textbf{B})$ satisfies this condition.

3 Caution when structuring the interference mechanism

Assumptions restricting the interference mechanism (usually involving a notion of locality) can alleviate some of the issues mentioned in the previous section (Hudgens and Halloran, 2012; Sinclair et al., 2012; Basse and Feller, 2016; Ugander et al., 2013), and allow the existence of unbiased estimators for a variety of estimands (Aronow and Samii, 2013; Ugander et al., 2013). Even then, however, basic intuition from the no-interference case remains misleading. When considering such assumptions, it is important to assess their impact on both the bias and the variance of estimators, especially in settings which naturally afford large sample sizes, where it is tempting to focus mostly on the bias, and ignore potential issues with the variance. We illustrate this danger with a realistic example in which a local interference assumption leads to an estimator that is unbiased, but whose variance explodes in large sample settings (Aronow and Samii, 2013; Ugander et al., 2013).

If $d(i,j)$ is a measure of the distance between units $i$ and $j$, it is often plausible to assume that only units in the k-step neighborhood $\mathcal{N}_{i}^{(k)}=\{j=1\ldots N:d(i,j)\leq k\}$ of $i$ can interfere with its outcome (Ugander et al., 2013; Coppock and Sircar, 2013). Formally,

Assumption 3** (k-local interference).**

[TABLE]

where $\textbf{Z}_{\mathcal{N}_{i}^{(k)}}$ denotes the sub vector of Z containing the assignments of the k-step neighbors of unit $i$. This assumption gives a special role to the following subsets of $\mathcal{Z}$:

[TABLE]

where $\mathcal{Z}_{i}^{(k)}(\textbf{B})$ is defined similarly. The set $\mathcal{Z}_{i}^{(k)}(\textbf{A})$ is the set of assignments in which unit $i$ and all its $k-step$ neighbors are assigned to $A$. Consider the following Horvitz-Thompson estimator for the average causal effect:

[TABLE]

which can be shown to be unbiased under $k-local$ interference (Aronow and Samii, 2013). Under the Bernoulli Design, the variance has the following expression:

Proposition 2**.**

[TABLE]

where:

[TABLE]

and similarly for $\mathbb{V}_{\mathcal{R}}[\hat{\theta}_{\textbf{B}}]$, and:

[TABLE]

The variance of the unbiased estimator thus depends explicitly on network quantities. To make things even more explicit, the next example focuses on a specific family of networks and shows that whether the estimator is consistent depends on a single parameter of the network family:

Example 3**.**

For this example, we will assume that:

[TABLE]

and will focus on the case where $k=1$. If we model the network as an Erdos-Renyi graph with probability $p$ of connection between nodes, we can show that:

[TABLE]

The behavior of this quantity depends on the parameter $p$, which governs the sparsity of the network. We show in appendix that

• •

if $p<\frac{1}{N}$, we have: \mathbb{E}_{\mathcal{G}}\bigg{[}\mathbb{V}_{\mathcal{R}}[\hat{\theta}]\bigg{]}\leq O(1/N),

• •

if $p\geq 1/\sqrt{N}$, we have: \mathbb{E}_{\mathcal{G}}\bigg{[}\mathbb{V}_{\mathcal{R}}[\hat{\theta}]\bigg{]}\geq\frac{4e^{\frac{N-1}{\sqrt{N}}}}{N}K^{2},

In this case, the expected variance of the estimator goes to zero if $p<1/N$, but not if $p\geq 1/\sqrt{N}$.

It is straightforward to construct unbiased estimators under most forms of localized interference, by relying on the popular Horvitz-Thompson estimators (Aronow and Samii, 2013). On the other hand, checking that the variance of such estimators converges – even under simple forms of interference – can be difficult. Yet, Example 3 shows the perils of neglecting this arduous task, especially in large samples.

4 Discussion

4.1 Broader class of estimands

The estimands of the form $\theta(\textbf{A},\textbf{B})$, which we have considered in this article, are relevant whenever the purpose of the experiment is to decide which of treatment A or treatment B would be best if applied to the entire population. This is the kind of question social platforms care about when they experiment with new products or features (Eckles et al., 2016; Gui et al., 2015). Other scenarios, however, would call for different kinds of estimands. In epidemiology for instance, the question of interest is often not which of two vaccines would be better if applied to the whole population, but which proportion of the population should be vaccinated (Hudgens and Halloran, 2012). It should be clear that the fundamental problems raised by arbitrary interference don’t vanish when more complex estimands are considered, although results analogous to Section 2 do become harder to formulate as we illustrate next. For the rest of this section, consider treatment A to be an “active treatment”, while treatment B is a “control” or ”no treatment”, and define the average primary causal effect:

[TABLE]

We can state the equivalent of Proposition 4:

Proposition 3**.**

Denote by $\textbf{Z}^{(i)}$ the assignment such that $Z_{i}=A$ and $Z_{j}=B$ for all $j\neq i$. Under arbitrary interference, the only unbiased estimators of $\theta$ under the Bernoulli Design are of the form:

[TABLE]

where $\sum_{\textbf{Z}}C(\textbf{Z})=0$.

The statement of Proposition 3 is more complex, but the problems it highlights are identical. Similar lines of reasoning hold for more complex estimands under arbitrary interference.

4.2 Effective treatments and informative sets

In Section 3, the outcome of unit $i$ depends on the assignment of units in its k-step neighborhood $\mathcal{N}_{i}^{(k)}$. This concept can be generalized by defining the reference group of user $i$ (Manski, 2013) to be the smallest set of units $G_{i}\subset\{1,\ldots,N\}$ such that:

[TABLE]

This generalizes Equation 2 and suggests that $Z_{G_{i}}$, called the effective treatment of unit $i$ (Manski, 2013; Aronow and Samii, 2013), is more relevant than its treatment $Z_{i}$. Under Assumption 2 (no interference) the treatment and effective treatment of unit $i$ are the same $\textbf{Z}_{G_{i}}=Z_{i}$, while under under Assumption 3, the effective treatment of unit $i$ encompasses the assignment of its k-step neighbors $\textbf{Z}_{G_{i}}=\textbf{Z}_{\mathcal{N}_{i}^{(k)}}$. Under arbitrary interference the effective treatment of unit $i$ is the entire assignment vector, $\textbf{Z}_{G_{i}}=\textbf{Z}$. In the language of Section 2, an assignment vector $\textbf{Z}^{\prime}$ will be informative for the outcome $Y_{i}(\textbf{Z})$ if it results in the same effective treatment. We thus define the informative set for $Y_{i}(\textbf{Z})$:

[TABLE]

Under Assumption 2, the informative set for any outcome $Y_{i}(\textbf{Z})$ contains every assignments $\textbf{Z}^{\prime}$ such that $Z^{\prime}_{i}=Z_{i}$, while under arbitrary interference, its informative set only contains the assignment Z. The next section explores how the concepts of effective treatment and informative sets capture the important changes implied by different interference structures.

4.3 Understanding interference structures

Although we have defined effective treatments and informative sets for abstract designs $\mathcal{R}$, we focus the rest of the discussion on the Bernoulli Design which captures the salient features of the problem, while simplifying the exposition. Denote by $E_{i}=|\{\textbf{Z}_{G_{i}}\}_{\textbf{Z}}|$ the number of effective treatments for unit $i$ and $S_{i}(\textbf{Z})=|\mathcal{Z}_{i}(\textbf{Z})|$ the size of the informative set for the outcome $Y_{i}(\textbf{Z})$. We call $F_{i}(\textbf{Z})=S_{i}(\textbf{Z})/|\mathcal{Z}|$ the fraction of informative assignments for $Y_{i}(\textbf{Z})$. Under the Bernoulli Design, we show in Appendix that:

[TABLE]

establishing a connection between the number of effective treatments and the fraction of informative assignments for all units $i$. Table 1 illustrates this relation for different interference structures.

We see that stronger assumptions on the interference structure tend to reduce the number of effective treatments or, equivalently, increase the fraction of relevant sets. Applying these insights to Example 3, Table 2 contrasts the behaviors of $\mathbb{E}_{\mathcal{G}}[E_{i}]$ and $\mathbb{E}_{\mathcal{G}}[F_{i}(\textbf{Z})]$ as $N\rightarrow\infty$ for sparser networks ($p=1/N$) and denser networks ($p=1/\sqrt{N}$). For sparser networks, the expected number of effective treatments converges while the fraction of informative assignments converges to a strictly positive number. In contrast, denser networks ($p=1/\sqrt{N}$) lead to an infinite number of effective treatments in expectation, as $N\rightarrow\infty$.

Under assumption 3, a unit’s outcome depends on the the assignment of its neighbors and so the fewer neighbors it has, the closest it is to the no-interference scenario. This intuition is supported by noticing that the column of Table 2 corresponding to the sparser networks ($p=1/N$) is closer to the column of Table 1 corresponding to the no-interference case, while the denser networks are closer to the arbitrary interference case. Although none of the observations we made describe sufficient conditions for the consistency of our estimator (see Aronow and Samii (2013)), they provide an intuitive connection between its asymptotic variance and the number of effective treatments (or equivalently, the fraction of informative assignments).

4.4 Connection with the Incidental Parameter Problem

The focus of this paper is on design-based inference: we have considered the potential outcomes as being fixed, and the randomness as coming from the assignment mechanism exclusively. In this section only, we will switch viewpoints and model the potential outcomes. This alternate perspective which statisticians tend to be more familiar with will hopefully provides additional insights into the problem.

Under the assumption of no interference, one might model the potential outcomes as follows:

[TABLE]

where to simplify, we assume that $\Sigma=I$. This way, there are only two parameters $\mu_{A}$ and $\mu_{B}$ to be estimated, regardless of the sample size $N$. When N increases, we have more and more units to estimate the same number of parameters (assuming that the fraction of units in each treatment remains constant). This is the traditional asymptotic regime considered in statistics.

If we allow arbitrary interference, however, the model naturally becomes:

[TABLE]

where we make the same simplification as above and assume $\Sigma=I$. The only unknowns are now the elements of $\vec{\mu}_{\mathcal{Z}}$, which is a vector of length $2^{N}$. This means that the number of parameters to estimate grows exponentially with the size of the problem. This scenario is sometimes referred to as the incidental parameter problem (Neyman and Scott, 1948), and the challenges it poses are well known.

Appendix A Technical appendix

A.1 Variance terms in Section 2

[TABLE]

A.2 Proof of theorems

Theorem 3**.**

Consider any estimand $\theta(\textbf{A},\textbf{B})=g(\textbf{Y}(\textbf{A}),\textbf{Y}(\textbf{B}))$ that is not constant, and any assignment mechanism $\mathcal{R}$ such that $\mathbb{P}_{\mathcal{R}}(\textbf{Z}=\textbf{A})=\mathbb{P}_{\mathcal{R}}(\textbf{Z}=\textbf{B})=0$. There exists no unbiased estimator of $\theta(\textbf{A},\textbf{B})$ under $\mathcal{R}$.

Proof.

The key insight behind the proof is that under arbitrary interference, if Z and $\textbf{Z}^{\prime}$ are two assignments such that $\textbf{Z}\neq\textbf{Z}^{\prime}$, then $Y(\textbf{Z})$ and $Y(\textbf{Z}^{\prime})$ can be seen as two independent parameters. That is, the value of $Y(\textbf{Z})$ does not constrain that of $Y(\textbf{Z}^{\prime})$. Denote by $p(\textbf{Z}_{0})$ the probability $\mathbb{P}_{\mathcal{R}}(\textbf{Z}=\textbf{Z}_{0})$, and assume that there exists an unbiased estimator $\hat{g}(Y(\textbf{Z}))$. This means that:

[TABLE]

since $p(\textbf{A})=p(\textbf{B})=0$. Now the LHS does not depend on the independent parameters $Y(\textbf{A})$ or $Y(\textbf{B})$, and so the RHS must not depend on $Y(\textbf{A})$ or $Y(\textbf{B})$ either. But $RHS$ is only a function of these two parameters, so we must have: $g(Y(\textbf{A}),Y(\textbf{B}))=C$, where $C$ is a constant. This violates the hypothesis that $g$ is not degenerate. Both CRD and CRB satisfy $p(\textbf{A})=p(\textbf{B})=0$, so the result holds for both designs. ∎

Proposition 4**.**

Consider the Bernoulli Design. If the estimand is of additive form $\theta(\textbf{A},\textbf{B})=\theta_{1}(\textbf{A})+\theta_{2}(\textbf{B})$, then unbiased estimators are of the form

[TABLE]

where $C(\textbf{Z})$ does not depend on any potential outcomes and satisfies:

[TABLE]

For other types of estimands $\theta(\textbf{A},\textbf{B})$, there exist no unbiased estimators.

Proof.

Recall that we use the notation $\theta_{1}(\textbf{A})=g_{1}(\textbf{Y}(\textbf{A}))$ and $\theta_{2}(\textbf{B})=g_{2}(\textbf{Y}(\textbf{B}))$. Let $\mathcal{R}$ be the Bernoulli Design. Suppose that $\hat{\theta}(\textbf{Z})=\hat{g}(\textbf{Y}(\textbf{Z}))$ is unbiased for $g(\textbf{Y}(\textbf{A}),\textbf{Y}(\textbf{B}))$. We have:

[TABLE]

Now RHS of the equation doesn’t depend on $\textbf{Y}(\textbf{A})$ or $\textbf{Y}(\textbf{B})$ so the LHS mustn’t either. Moreover, the LHS only depends on $\textbf{Y}(\textbf{A})$ and $\textbf{Y}(\textbf{B})$, so in the end, the LHS cannot depend on any potential outcome $\{Y(Z)\}_{Z}$. That is, we must have:

[TABLE]

and so:

[TABLE]

where $C$ does not depend on any potential outcome. But then this means that

[TABLE]

But $\hat{g}(\textbf{Y}(\textbf{B}))$ and $\hat{g}(\textbf{Y}(\textbf{A}))$ are functions of $\textbf{Y}(\textbf{B})$ and $\textbf{Y}(\textbf{A})$ respectively, so this means that if the estimand $g$ can be estimated unbiasedly, it has to be of the form:

[TABLE]

since $C^{*}$ doesn’t depend on any potential outcome (for instance, let $\theta_{1}(\textbf{A})=C^{*}+g_{1}(\textbf{Y}(\textbf{A}))$ and $\theta_{2}(\textbf{B})=g_{2}(\textbf{Y}(\textbf{B}))$…). Now consider any estimand of the form $g(\textbf{Y}(\textbf{A}),\textbf{Y}(\textbf{B}))=C^{*}+g_{1}(\textbf{Y}(\textbf{A}))+g_{2}(\textbf{Y}(\textbf{B}))$. Plugging into Equation 9, we have:

[TABLE]

and so:

[TABLE]

now we use the same reasoning as above, leading to the fact that:

[TABLE]

and

[TABLE]

Return now to Equation 8, focusing on the RHS and let $\textbf{Z}_{0}\not\in\{\textbf{A},\textbf{B}\}$. A similar argument as the one used above gives:

[TABLE]

using the same reasoning once again, we conclude that:

[TABLE]

where C does not depend on any potential outcome. And so putting it all together, we have:

[TABLE]

∎

Theorem 4**.**

Consider any estimand $\theta(\textbf{A},\textbf{B})=g(\textbf{Y}(\textbf{A}),\textbf{Y}(\textbf{B}))$, and suppose that $g$ is onto the interval [0, M]. That is,

[TABLE]

Then for all sample size $N$ and estimator $\hat{\theta}(\textbf{Z})$ there exist potential outcomes satisfying Assumption 1 such that:

[TABLE]

where the MSE is taken under Completely Randomized Design or Bernoulli Design .

Proof.

We write down the MSE:

[TABLE]

Consider potential outcomes $\{\textbf{Y}(\textbf{Z})\}_{Z}$ such that $\textbf{Y}(\textbf{Z})=\textbf{Y}=cst$ for all $\textbf{Z}\not\in\{\textbf{A},\textbf{B}\}$, and let $C=\hat{g}(\textbf{Y})$. We now need to consider separately the case of Bernoulli Design (BD) and Completely Randomized Design (CRD). We start with the CRD. Let $\mathcal{Z}_{N_{A}}$ be the set of all assignments with $N_{A}$ units assigned to treatment $A$. We have:

[TABLE]

Since g is surjective, chose $\textbf{Y}(\textbf{A})$ and $\textbf{Y}(\textbf{B})$ such that:

[TABLE]

and so:

[TABLE]

∎

now let’s turn to the BD case. The slight difference is that the assignments $\textbf{Z}=\textbf{A}$ and $\textbf{Z}=\textbf{B}$ may occur. We thus have:

[TABLE]

which concludes the proof.

Remark 2**.**

The surjective assumption is not crucial to the spirit of the result, but it does make it easier to state the theorem, and highlight the magnitude of how much things can go wrong. Note that the popular average total effect estimand satisfies the conditions of the theorem.

Proposition 5**.**

[TABLE]

where:

[TABLE]

and similarly for $\mathbb{V}_{\mathcal{R}}[\hat{\theta}_{\textbf{B}}]$, and:

[TABLE]

Proof.

Throughout, we will let $I_{i}(\textbf{A})=I(\textbf{Z}\in\mathcal{Z}_{i}^{(k)}(\textbf{A}))$ and $P_{i}(\textbf{A})=P(I_{i}(\textbf{A})=1)$. We begin by looking at $\mathbb{V}_{\mathcal{R}}(\hat{\theta}_{A})$. We have:

[TABLE]

But we also have:

[TABLE]

and:

[TABLE]

where P_{ij}(\textbf{A},\textbf{A})=P\bigg{(}I_{i}(\textbf{A})=1,I_{j}(\textbf{A})=1\bigg{)}. And so finally:

[TABLE]

But under k-local exposure and Bernoulli Design, we have P_{i}(\textbf{A})=\bigg{(}\frac{1}{2}\bigg{)}^{|\mathcal{N}_{i}^{(k)}|} and

[TABLE]

and so, plugging into the equation for $\mathbb{V}_{\mathcal{R}}(\hat{\theta}_{A})$, we have.

[TABLE]

We now turn to the covariance term $\mathbb{C}ov_{\mathcal{R}}(\hat{\theta}_{A},\hat{\theta}_{B})$. We have:

[TABLE]

but we have:

[TABLE]

and

[TABLE]

and so putting it all together, we have:

[TABLE]

which completes the proof, since $1-I(|\mathcal{N}^{(k)}_{i}\cap\mathcal{N}^{(k)}_{j}|=0)=I(|\mathcal{N}^{(k)}_{i}\cap\mathcal{N}^{(k)}_{j}|>0)$.

∎

Proposition 6**.**

Denote by $\textbf{Z}^{(i)}$ the assignment such that $Z_{i}=A$ and $Z_{j}=B$ for all $j\neq i$. Under arbitrary interference, the only unbiased estimators of $\theta=\frac{1}{N}\sum_{i}Y_{i}(Z_{i}=A,\textbf{Z}_{-i}=\textbf{B})$ under the Bernoulli Design are of the form:

[TABLE]

where $\sum_{\textbf{Z}}C(\textbf{Z})=0$.

Proof.

The proof follows the same lines as that of Proposition 1. Let $\hat{\theta}(\textbf{Z})=\hat{g}(\textbf{Y}(\textbf{Z}))$ be an unbiased estimator of $\theta$ under the $BernoulliDesign$. Thus we have:

[TABLE]

The LHS does not depend on any $\textbf{Y}(\textbf{Z}^{(i)})$ for $i=1,\ldots,N$ so the RHS mustn’t either. But the RHS only depends on $\textbf{Y}(\textbf{Z}^{(i)})$ for $i=1,\ldots,N$, so it must be that $RHS$ doesn’t depend on any potential outcome. That is,

[TABLE]

where $C(\textbf{Z})$ does not depend on any potential outcomes. Now consider an index $i_{0}$. We have, using the previous equation:

[TABLE]

Again, the LHS does not depend on $\textbf{Y}(\textbf{Z}^{(i_{0})})$ so the RHS shouldn’t either. But the RHS depends explicitly on $\textbf{Y}(\textbf{Z}^{(i_{0})})$ , so it must be that the RHS doesn’t depend on any potential outcomes. Hence:

[TABLE]

That is, renaming $C^{*}(\textbf{Z})\rightarrow C(\textbf{Z})$, we have:

[TABLE]

Now we could apply the same reasoning to any index $i$, so:

[TABLE]

Now going back to the first equation of the proof, the RHS does not depend on any potential outcomes either. So for any $\textbf{Z}_{0}\in\mathcal{Z}-\cup_{i}\textbf{Z}^{(i)}$:

[TABLE]

here again, the $LHS$ does not depend on $\textbf{Y}(\textbf{Z}_{0})$ so the $RHS$ shouldn’t either. But the RHS depends explicitly only on $\textbf{Y}(\textbf{Z}_{0})$, and so it must be that it depends on no potential outcomes. Hence:

[TABLE]

Finally putting it all together we have:

[TABLE]

this can be rewritten:

[TABLE]

∎

A.3 Derivations for Example 3

We now assume that $k=1$, and the networks are $ER(N,p)$. We also assume that there exists $K,M>0$ such that:

[TABLE]

Define:

[TABLE]

Since all the quantities are positive, it is easy to verify that:

[TABLE]

Under the assumptions stated for this section, we have:

[TABLE]

where $d_{i}$ is the degree of node $i$. Under the $ER(N,p)$ model, we have $d_{i}\sim Bin(N-1,p)$. It is then easy to verify (via the moment generating function) that:

[TABLE]

and so:

[TABLE]

Similarly, we have:

[TABLE]

where $I_{ij}\sim Bern(p)$, $C_{ij}\sim Bin(N-2,p^{2})$, and $I_{ij}\perp C_{ij}$. So we have:

[TABLE]

and:

[TABLE]

and so

[TABLE]

In fact, we will augment this notation a bit, and denote it by $h_{N}(C,p)$, to mark the dependence on $C$ and on $p$. Note that we can verify that $h_{N}(C,p)$ is monotone non-decreasing in $p$.

Sparse graphs: $p<\frac{1}{N}$:

Let $p_{N}=\frac{1}{N}$, we have:

[TABLE]

and so:

[TABLE]

Similarly, we have:

[TABLE]

and finally, we have:

[TABLE]

And so putting it all together, we have $h_{N}(M,p_{N})=O(\frac{1}{N})$ and so finally:

[TABLE]

Dense graphs: $p>\frac{1}{\sqrt{N}}$:

Let $p_{N}=\frac{1}{\sqrt{N}}$. We have:

[TABLE]

and so:

[TABLE]

A.4 Derivations for Section 4

$E_{i}$ , $S_{i}(\textbf{Z})$ and $F_{i}$ under BD:

We have seen that under no-interference, $G_{i}=\{i\}$ so the effective treatments are $\{\textbf{Z}_{\{G_{i}\}}\}_{\textbf{Z}\in\mathcal{Z}}=\{A,B\}$ and so $E_{i}=2$. Under arbitrary interference, $G_{i}=\{1,\ldots,N\}$, so the effective treatments are $\{\textbf{Z}_{\{G_{i}\}}\}_{\textbf{Z}\in\mathcal{Z}}=\{\textbf{Z}\}_{\textbf{Z}\in\mathcal{Z}}$ and so $E_{i}=2^{N}$. Finally, under 1-local interference, $G_{i}=\mathcal{N}_{i}$, so $\{\textbf{Z}_{\{G_{i}\}}\}_{\textbf{Z}\in\mathcal{Z}}=\{\textbf{Z}_{\mathcal{N}_{i}}\}_{\textbf{Z}\in\mathcal{Z}}$, and thus $E_{i}=2^{|\mathcal{N}_{i}|}$ under BD.

Recall that $S_{i}(\textbf{Z})=|\mathcal{Z}_{i}(\textbf{Z})|$, where:

[TABLE]

Under the Bernoulli Design and 1-local interference $G_{i}=\mathcal{N}_{i}$. So if we denote by $\textbf{Z}_{-\mathcal{N}_{i}}$ the sub-vector of Z excluding the neighborhood of $i$, $\textbf{Z}_{-\mathcal{N}_{i}}$ is of dimension $N-\mathcal{N}_{i}$ and

[TABLE]

it immediately follows that $F_{i}=\frac{1}{E_{i}}$.

$\mathbb{E}_{\mathcal{G}}(E_{i})$ and $\mathbb{E}_{\mathcal{G}}(S_{i}(\textbf{Z}))$ under BD:

We have:

[TABLE]

and so for $p=1/N$ we have:

[TABLE]

and so the fraction of all assignments that correspond to effective treatment of $i$ is asymptotically:

[TABLE]

in the dense case however, we have:

[TABLE]

and so asymptotically:

[TABLE]

Similarly, we have:

[TABLE]

and so for $p=1/N$, we have:

[TABLE]

while for $p=1/\sqrt{N}$ we have:

[TABLE]