Inference on Breakdown Frontiers

TL;DR

This paper introduces the concept of breakdown frontiers to assess the robustness of causal conclusions in treatment effect analysis, providing estimators and confidence bands to quantify how assumptions affect inferences.

Contribution

It develops a framework for deriving and estimating breakdown frontiers under relaxations of key assumptions in potential outcomes models, with practical bootstrap methods for inference.

Findings

Robustness of conclusions varies with assumption relaxations.

Estimated frontiers quantify how assumptions impact causal inferences.

Application shows some conclusions are robust to assumption failures.

Abstract

Given a set of baseline assumptions, a breakdown frontier is the boundary between the set of assumptions which lead to a specific conclusion and those which do not. In a potential outcomes model with a binary treatment, we consider two conclusions: First, that ATE is at least a specific value (e.g., nonnegative) and second that the proportion of units who benefit from treatment is at least a specific value (e.g., at least 50\%). For these conclusions, we derive the breakdown frontier for two kinds of assumptions: one which indexes relaxations of the baseline random assignment of treatment assumption, and one which indexes relaxations of the baseline rank invariance assumption. These classes of assumptions nest both the point identifying assumptions of random assignment and rank invariance and the opposite end of no constraints on treatment selection or the dependence structure between potential outcomes. This frontier provides a quantitative measure of robustness of conclusions to relaxations of the baseline point identifying assumptions. We derive $\sqrt{N}$-consistent sample analog estimators for these frontiers. We then provide two asymptotically valid bootstrap procedures for constructing lower uniform confidence bands for the breakdown frontier. As a measure of robustness, estimated breakdown frontiers and their corresponding confidence bands can be presented alongside traditional point estimates and confidence intervals obtained under point identifying assumptions. We illustrate this approach in an empirical application to the effect of child soldiering on wages. We find that sufficiently weak conclusions are robust to simultaneous failures of rank invariance and random assignment, while some stronger conclusions are fairly robust to failures of rank invariance but not necessarily to relaxations of random assignment.

Figures

Tables

Table 1. Table 1: Summary statistics

Variable Name	Mean	Median	Stddev	Min	Max
Daily wage in Uganda shillings	2957.60	1400.00	6659.76	35.71	83333.34
Log wage	7.23	7.24	1.18	3.58	11.33
Not abducted?	0.36	0.00	0.48	0.00	1.00
Age when surveyed	22.11	22.00	4.88	14.00	30.00
Household size in 1996	8.31	8.00	4.19	2.00	28.00
Sample size is 448. 1 USD is approximately 1800 Uganda shillings (Exchange rate at time of survey, 2005-2006; source: World Bank).

Table 2. Table 2: Comparison of means

Variable Name	Not Abducted	Abducted	Difference
Daily wage in Uganda shillings	3409.12	2706.75	702.36   [725.49]
Log wage	7.33	7.18	0.15   [0.12]
Age when surveyed	21.23	22.60	-1.37   [0.48]
Household size in 1996	8.53	8.19	0.34   [0.42]
Observations	160	288
Sample size is 448. 1 USD is approximately 1800 Uganda shillings (Exchange rate at time of survey, 2005-2006; source: World Bank). Standard errors in brackets.

Equations

$$\mathbb{P}(Y_1>Y_0)\ge \underline{p}$$

$$\text{ATE}\ge \mu$$

$$Y=X{Y}_1+(1-X)Y_0.$$

$$\underset{y\in \mathrm{supp}(Y_x\mid W=w)}{\sup }|\mathbb{P}(X=x\mid Y_x=y,W=w)-\mathbb{P}(X=x\mid W=w)|\le c$$

$$F_{Y_1,Y_0\mid W}(y_1,y_0\mid w)=C(F_{Y_1\mid W}(y_1\mid w),F_{Y_0\mid W}(y_0\mid w)\mid w)$$

$$C(u_1,u_0\mid w)=\min \{u_1,u_0\}\equiv M(u_1,u_0),$$

$$\max \{u_1+u_0-1,0\}\le C(u_1,u_0\mid w)\le M(u_1,u_0).$$

$$C(u_1,u_0\mid w)=(1-t)M(u_1,u_0)+tH(u_1,u_0\mid w)$$

$$F_{Y_1,Y_0\mid W}(y_1,y_0\mid w)$$

$$=(1-t)F_{Y_1,Y_0\mid W,T}(y_1,y_0\mid w,0)+t{F}_{Y_1,Y_0\mid W,T}(y_1,y_0\mid w,1)$$

$$\displaystyle=(1-t)C\big{(}F_{Y_{1}\mid W,T}(y_{1}\mid w,0),F_{Y_{0}\mid W,T}(y_{0}\mid w,0)\mid w,0\big{)}$$

$$\displaystyle\qquad+t\,C\big{(}F_{Y_{1}\mid W,T}(y_{1}\mid w,1),F_{Y_{0}\mid W,T}(y_{0}\mid w,1)\mid w,1\big{)}$$

$$=(1-t)M(F_{Y_1\mid W}(y_1\mid w),F_{Y_0\mid W}(y_0\mid w))+tH(F_{Y_1\mid W}(y_1\mid w),F_{Y_0\mid W}(y_0\mid w)\mid w).$$

$$\displaystyle\Big{[}\underline{\text{CQTE}}(\tau,c\mid w),\overline{\text{CQTE}}(\tau,c\mid w)\Big{]}$$

$$\equiv \left[{\underline{Q}}_{Y_1\mid W}^c(\tau \mid w)-{\overline{Q}}_{Y_0\mid W}^c(\tau \mid w),{\overline{Q}}_{Y_1\mid W}^c(\tau \mid w)-{\underline{Q}}_{Y_0\mid W}^c(\tau \mid w)\right]$$

$${\overline{Q}}_{Y_x\mid W}^c(\tau \mid w)$$

$${\underline{Q}}_{Y_x\mid W}^c(\tau \mid w)$$

$$[\underline{\text{CATE}}(c\mid w),\overline{\text{CATE}}(c\mid w)]\equiv \left[{\int }_0^1\underline{\text{CQTE}}(\tau ,c\mid w)d\tau ,{\int }_0^1\overline{\text{CQTE}}(\tau ,c\mid w)d\tau \right]$$

$${\overline{F}}_{Y_x\mid W}^c(y\mid w)=\min \left\{\frac{F_{Y\mid X,W}(y\mid x,w)p_{x|w}}{p_{x|w}-c},\frac{F_{Y\mid X,W}(y\mid x,w)p_{x|w}+c}{p_{x|w}+c}\right\}$$

$${\underline{F}}_{Y_x\mid W}^c(y\mid w)=\max \left\{\frac{F_{Y\mid X,W}(y\mid x,w)p_{x|w}}{p_{x|w}+c},\frac{F_{Y\mid X,w}(y\mid x,w)p_{x|w}-c}{p_{x|w}-c}\right\}$$

$$\text{DTE}(z)=\mathbb{P}(Y_1-Y_0\le z).$$

$$\text{CDTE}(z\mid w)=\mathbb{P}(Y_1-Y_0\le z\mid W=w).$$

$$\text{DTE}(z)=\mathbb{E}[\text{CDTE}(z\mid W)].$$

$$\begin{gathered} \overline{\text{CDTE}}(z,c,t\mid w)=(1-t)\mathbb{P}({\underline{Q}}_{Y_1\mid W}^c(U\mid w)-{\overline{Q}}_{Y_0\mid W}^c(U\mid w)\le z) \\ +t\left(1+\min \left\{\underset{y\in {\mathcal{Y}}_z(w)}{\inf }({\overline{F}}_{Y_1\mid W}^c(y\mid w)-{\underline{F}}_{Y_0\mid W}^c(y-z\mid w)),0\right\}\right) \end{gathered}$$

$$\begin{gathered} \underline{\text{CDTE}}(z,c,t\mid w)=(1-t)\mathbb{P}({\overline{Q}}_{Y_1\mid W}^c(U\mid w)-{\underline{Q}}_{Y_0\mid W}^c(U\mid w)\le z) \\ +t\max \left\{\underset{y\in {\mathcal{Y}}_z(w)}{\sup }({\underline{F}}_{Y_1\mid W}^c(y\mid w)-{\overline{F}}_{Y_0\mid W}^c(y-z\mid w)),0\right\} \end{gathered}$$

$$[\underline{\text{CDTE}}(z,c,t\mid w),\overline{\text{CDTE}}(z,c,t\mid w)].$$

$$\begin{gathered} [\underline{\text{DTE}}(z,c,t),\overline{\text{DTE}}(z,c,t)] \\ =\left[{\int }_{\mathrm{supp}(W)}\underline{\text{CDTE}}(z,c,t\mid w)d{F}_W(w),{\int }_{\mathrm{supp}(W)}\overline{\text{CDTE}}(z,c,t\mid w)d{F}_W(w)\right]. \end{gathered}$$

$$\mathbb{P}(Y_1-Y_0>z)\in [1-\overline{\text{DTE}}(z,c,t),1-\underline{\text{DTE}}(z,c,t)].$$

$$\text{RR}(z,\underline{p})$$

$$=\{(c,t)\in [0,1{]}^2:\overline{\text{DTE}}(z,c,t)\le 1-\underline{p}\}.$$

$$\text{BF}(\underline{p})=\{(c,t)\in [0,1{]}^2:\overline{\text{DTE}}(0,c,t)=1-\underline{p}\}.$$

$$\text{bf}(c,\underline{p})=\frac{\text{num}}{\text{denom}}$$

$$\text{num}=1-\underline{p}-{\int }_{\mathrm{supp}(W)}\mathbb{P}({\underline{Q}}_{Y_1\mid W}^c(U\mid w)-{\overline{Q}}_{Y_0\mid W}^c(U\mid w)\le 0)d{F}_W(w)$$

$$\text{denom}=1+\int_{\operatorname*{supp}(W)}\bigg{[}\min\left\{\inf_{y\in\mathcal{Y}_{0}(w)}(\overline{F}^{c}_{Y_{1}\mid W}(y\mid w)-\underline{F}^{c}_{Y_{0}\mid W}(y-0\mid w)),0\right\}\\ -\mathbb{P}(\underline{Q}^{c}_{Y_{1}\mid W}(U\mid w)-\overline{Q}^{c}_{Y_{0}\mid W}(U\mid w)\leq 0)\bigg{]}\,dF_{W}(w).$$

Full text

JEL classification: C14; C18; C21; C25; C51

Keywords: Nonparametric Identification, Partial Identification, Sensitivity Analysis, Selection on Unobservables, Rank Invariance, Treatment Effects, Directional Differentiability

1 Introduction

Traditional empirical analysis combines the observed data with a set of assumptions to draw conclusions about a parameter of interest. Breakdown frontier analysis reverses this ordering. It begins with a fixed conclusion and a set of baseline assumptions and asks, ‘What are the weakest assumptions needed to draw that conclusion, given the observed data?’ For example, consider the impact of a binary treatment on some outcome variable. The traditional approach might assume random assignment, point identify the average treatment effect (ATE), and then report the obtained value. The breakdown frontier approach instead begins with a conclusion about ATE, like ‘ATE is positive’, and reports the weakest assumption—relative to random assignment—on the relationship between treatment assignment and potential outcomes needed to obtain this conclusion, when such an assumption exists. When more than one kind of assumption is considered, this approach leads to a curve, representing the weakest combinations of assumptions which lead to the desired conclusion. This curve is the breakdown frontier.

At the population level, the difference between the traditional approach and the breakdown frontier approach is a matter of perspective: an answer to one question is an answer to the other. This relationship has long been present in the literature initiated by Manski on partial identification (for example, see Manski 2007 or section 3 of Manski 2013). In finite samples, however, which approach one chooses has important implications for how one does statistical inference. Specifically, the traditional approach estimates the parameter or its identified set. Here we instead estimate the breakdown frontier. The traditional approach then performs inference on the parameter or its identified set. Here we instead perform inference on the breakdown frontier. Thus the breakdown frontier approach puts the weakest assumptions necessary to draw a conclusion at the center of attention. Consequently, by construction, this approach avoids the non-tight bounds critique of partial identification methods (for example, see section 7.2 of Ho and Rosen 2017). One distinction is that the traditional approach may require inference on a partially identified parameter. The breakdown frontier approach, however, only requires inference on a point identified object.

The breakdown frontier we study generalizes the concept of an “identification breakdown point” introduced by Horowitz and Manski (1995), a one dimensional breakdown frontier.333The identification breakdown point is distinct from the breakdown point introduced earlier by Hampel (1968, 1971) in the robust statistics literature that began with Huber (1964); also see Donoho and Huber (1983). Horowitz and Manski (1995) give a detailed comparison of the two concepts. Throughout this paper we use the term “breakdown” in the same sense as Horowitz and Manski’s identification breakdown point. Their breakdown point was further studied and generalized by Stoye (2005, 2010). Our emphasis on inference on the breakdown frontier follows Kline and Santos (2013), who proposed doing inference on a breakdown point. Finally, our focus on multi-dimensional frontiers builds on the graphical sensitivity analysis of Imbens (2003) and the multi-dimensional sensitivity analysis of Manski and Pepper (2018). We discuss these papers and others in detail in appendix A.

The breakdown frontier approach

The breakdown frontier approach requires six main steps: (a) specify a parameter of interest, (b) specify a set of baseline assumptions, (c) define a class of assumptions indexed by a sensitivity parameter which deliver a nested sequence of identified sets, with the baseline assumptions obtained at one extreme and the no assumptions bounds obtained at the other, (d) characterize identified sets for the parameter of interest as a function of the sensitivity parameter, (e) use those identified sets to define the breakdown frontier for a conclusion of interest, and (f) develop estimation and inference procedures for that frontier based on its characterization.

In principle this analysis can be done for a general class of models, for example, by using the general identification analysis in Chesher and Rosen (2017) or Torgovitsky (2018) for step (d), and then applying general tools for nonparametric estimation and inference for step (f). While such a general analysis is an important next step for future work, in this paper we focus on just one important and widely used model: the potential outcomes model with a binary treatment. By focusing on a single concrete model we can clearly illustrate how to do the six main steps (a)–(f) required for a breakdown frontier analysis in any model. While the mathematical analysis will differ from model to model, the general ideas and approach do not.

In the rest of this section and this paper, we use the binary treatment potential outcomes model to explain and illustrate the breakdown frontier approach. Our main parameter of interest is the proportion of units who benefit from treatment. Under random assignment of treatment and rank invariance, this parameter is point identified. One may be concerned, however, that these two assumptions are too strong. We relax rank invariance by supposing that there are two types of units in the population: one type for which rank invariance holds and another type for which it may not. The proportion $t$ of the second type measures the relaxation of rank invariance. We relax random assignment using a propensity score distance $c\geq 0$ as in our previous work, Masten and Poirier (2018a). We give more details on both of these relaxations in section 2. We derive the identified set for $\mathbb{P}(Y_{1}>Y_{0})$ as a function of $(c,t)$. For a specific conclusion, such as $\mathbb{P}(Y_{1}>Y_{0})\geq 0.5$, this identification result defines a breakdown frontier.

Figure 1 illustrates this breakdown frontier. The horizontal axis measures $c$, the relaxation of the random assignment assumption. The vertical axis measures $t$, the relaxation of rank invariance. The origin represents the baseline point identifying assumptions of random assignment and rank invariance. Points along the vertical axis represent random assignment paired with various relaxations of rank invariance. Points along the horizontal axis represent rank invariance paired with various relaxations of random assignment. Points in the interior of the box represent relaxations of both assumptions. The points in the lower left region are pairs of assumptions $(c,t)$ such that the data allow us to draw our desired conclusion: $\mathbb{P}(Y_{1}>Y_{0})\geq 0.5$. We call this set the robust region. Specifically, no matter what value of $(c,t)$ we choose in this region, the identified set for $\mathbb{P}(Y_{1}>Y_{0})$ always lies completely above 0.5. The points in the upper right region are pairs of assumptions that do not allow us to draw this conclusion. For these pairs $(c,t)$ the identified set for $\mathbb{P}(Y_{1}>Y_{0})$ contains elements smaller than 0.5. The boundary between these two regions is precisely the breakdown frontier. The area under the breakdown frontier—the robust region—is a quantitative measure of robustness.

Figure 2a illustrates how the breakdown frontier changes as our conclusion of interest changes. Specifically, consider the conclusion that

[TABLE]

for five different values for $\underline{p}$. The figure shows the corresponding breakdown frontiers. As $\underline{p}$ increases towards one, we are making a stronger claim about the true parameter and hence the set of assumptions for which the conclusion holds shrinks. For strong enough claims, the claim may be refuted even with the strongest assumptions possible. Conversely, as $\underline{p}$ decreases towards zero, we are making progressively weaker claims about the true parameter, and hence the set of assumptions for which the conclusion holds grows larger.

Under the strongest assumptions of $(c,t)=(0,0)$, the parameter $\mathbb{P}(Y_{1}>Y_{0})$ is point identified. Let $p_{0,0}$ denote its value. The value $p_{0,0}$ is often strictly less than 1. In this case, any $\underline{p}\in(p_{0,0},1]$ yields a degenerate breakdown frontier: This conclusion is refuted under the point identifying assumptions. Even if $p_{0,0}<\underline{p}$, the conclusion $\mathbb{P}(Y_{1}>Y_{0})\geq\underline{p}$ may still be correct. This follows since, for strictly positive values of $c$ and $t$, the identified sets for $\mathbb{P}(Y_{1}>Y_{0})$ do contain values larger than $p_{0,0}$. But they also contain values smaller than $p_{0,0}$. Hence there do not exist any assumptions for which we can draw the desired conclusion.

The breakdown frontier is similar to the classical Pareto frontier from microeconomic theory: It shows the trade-offs between different assumptions in drawing a specific conclusion. For example, consider figure 1. If we are at the top left point, where the breakdown frontier intersects the vertical axis, then any relaxation of random assignment requires strengthening the rank invariance assumption in order to still be able to draw our desired conclusion. The breakdown frontier specifies the precise marginal rate of substitution between the two assumptions. If we can interpret the units of each relaxation separately, we can interpret trade-offs between them. We discuss these individual interpretations in section 2. It can sometimes be helpful to measure the two different relaxations in common units. This is not necessary, however. For example, in labor supply models agents trade off leisure hours for consumption, although time and quantities of goods are measured in fundamentally different units. Many common rates outside of economics, like kilometers per hour or beats per minute, also do not have common units.

We also study breakdown frontiers and derive asymptotic distributional results for the average treatment effect $\text{ATE}=\mathbb{E}(Y_{1}-Y_{0})$ and quantile treatment effects $\text{QTE}(\tau)=Q_{Y_{1}}(\tau)-Q_{Y_{0}}(\tau)$ for $\tau\in(0,1)$. Because the breakdown frontier analysis for these two parameters is nearly identical, we focus on ATE for brevity. Since ATE does not rely on the dependence structure between potential outcomes, assumptions regarding rank invariance do not affect the identified set. Hence, for a specific conclusion like $\text{ATE}\geq 0$, the breakdown frontiers are vertical lines. This was not the case for the breakdown frontier for claims about $\mathbb{P}(Y_{1}>Y_{0})$, where the assumptions on independence of treatment assignment and on rank invariance interact.

Figure 2b illustrates how the breakdown frontier for ATE changes as our conclusion of interest changes. Specifically, consider the conclusion that

[TABLE]

for five different values for $\mu$. The figure shows the corresponding breakdown frontiers. As $\mu$ increases, we are making a stronger claim about the true parameter and hence the set of assumptions for which the conclusion holds shrinks. For strong enough claims, the claim may be refuted even with the strongest assumptions possible. This happens when $\mathbb{E}(Y\mid X=1)-\mathbb{E}(Y\mid X=0)<\mu$. Conversely, as $\mu$ decreases, we are making progressively weaker claims about the true parameter, and hence the set of assumptions for which the conclusion holds grows larger.

Breakdown frontiers can be defined for many simultaneous claims. For example, figure 2c illustrates breakdown frontiers for the joint claim that $\mathbb{P}(Y_{1}>Y_{0})\geq\underline{p}$ and $\text{ATE}\geq 0$, for five different values of $\underline{p}$. Compare these plots to those of figures 2a and 2b. For small $\underline{p}$, many pairs $(c,t)$ lead to the conclusion $\mathbb{P}(Y_{1}>Y_{0})\geq\underline{p}$. But many of these pairs do not also lead to the conclusion $\mathbb{E}(Y_{1}-Y_{0})\geq 0$, and hence those pairs are removed. As $\underline{p}$ increases, however, the constraint that we also want to conclude $\text{ATE}\geq 0$ eventually does not bind. Similarly, if we look at the breakdown frontier solely for our claim about ATE, many points $(c,t)$ are included which are ruled out when we also wish to make our claim about $\mathbb{P}(Y_{1}>Y_{0})$. This is especially true as $\underline{p}$ gets larger.

Although we focus on one-sided claims like $\mathbb{P}(Y_{1}>Y_{0})\geq\underline{p}$, one may also be interested in the simultaneous claim that $\mathbb{P}(Y_{1}>Y_{0})\geq\underline{p}$ and $\mathbb{P}(Y_{1}>Y_{0})\leq\overline{p}$, where $\underline{p}<\overline{p}$. Similarly to the joint one-sided claims about $\mathbb{P}(Y_{1}>Y_{0})$ and ATE discussed above, the breakdown frontier for this two-sided claim on $\mathbb{P}(Y_{1}>Y_{0})$ can be obtained by taking the minimum of the two breakdown frontier functions for each one-sided claim separately. Equivalently, the area under the breakdown frontier for the two-sided claim is just the intersection of the areas under the breakdown frontiers for the one-sided claims.

Inference

As mentioned above, although identified sets are used in its definition, the breakdown frontier is always point identified. Hence for estimation and inference we use mostly standard nonparametric techniques. We first propose a consistent sample-analog estimator of the breakdown frontier. We then do inference via two asymptotically valid bootstrap procedures. Our main approach, discussed in section 3, relies on important work by Dümbgen (1993), Fang and Santos (2019), and Hong and Li (2018). We discuss an alternative approach in supplemental appendix D. In both approaches we construct one-sided lower uniform confidence bands for the breakdown frontier. A one-sided band is appropriate because our inferential goal is to determine the set of assumptions for which we can still draw our conclusion. If $\alpha$ denotes the nominal size, then approximately $100(1-\alpha)$% of the time, all elements $(c,t)$ which are below the confidence band lead to population level identified sets for the parameters of interest which allow us to conclude that our conclusions of interest hold; we also discuss a testing interpretation in supplemental appendix A. We examine the finite sample performance of our main approach in several Monte Carlo simulations.

An empirical illustration of the breakdown frontier approach

Finally, we use our results to examine the role of assumptions in determining the effects of child soldiering on wages, as studied in Blattman and Annan (2010). We illustrate how our results can be used as a sensitivity analysis within a larger empirical study. Specifically, we begin by first estimating and doing inference on parameters under point identifying assumptions. Next, we estimate breakdown frontiers for several claims about these parameters. We present our one-sided confidence bands as well. We then use these breakdown frontier point estimates and confidence bands to judge the sensitivity of our conclusion to the point identifying assumptions: A large inner confidence set for the robust region which includes many different points $(c,t)$ suggests robust results while a small inner confidence set close to the origin suggests non-robust results.

Motivation for the parameter $\mathbb{P}(Y_{1}>Y_{0})$

Our empirical illustration uses data from Blattman and Annan (2010), which is part of a large literature on the impact of compulsory military service on wages. As Card and Cardoso (2012, pages 57–58) note,

“Angrist (1990) showed that Vietnam-era draftees had lower earnings than non-draftees, a finding he attributed to the low value of military experience in the civilian labor market. Subsequent research in the United States and other countries, however, has uncovered a surprisingly mixed pattern of impacts.”

On one hand, enlistees learn basic skills and receive occupational training in the military. On the other hand, they forgo civilian schooling and work experience, and may experience debilitating psychological trauma. Which of these two explanations is correct? Most likely, both. By using models where the treatment effects $Y_{1}-Y_{0}$ are heterogeneous, we allow some people to have overall positive effects, and thus satisfy the first explanation, and some people to have overall negative effects, and thus satisfy the second explanation. The parameter $\mathbb{P}(Y_{1}>Y_{0})$ tells us precisely which proportion of the population primarily satisfies the first versus the second explanation. Thus it gives researchers a more nuanced understanding of treatment effects, by quantitatively measuring the prevalence of two opposing explanations of the impact of treatment on outcomes. We suspect this parameter can be particularly helpful in literatures which find a “mixed pattern of impacts,” like the work on compulsory military service and wages discussed above.

That said, one may be concerned that the rank invariance assumption used to point identify this parameter is too strong. As in Heckman et al. (1997), this concern motivates our sensitivity analysis. For further motivation of this parameter in various settings, see Bedoya et al. (2017) and Mullahy (2018).

2 Model and identification

To illustrate the breakdown frontier approach, we study the standard potential outcomes model with a binary treatment. We focus on two key assumptions: random assignment and rank invariance. We discuss how to relax these two assumptions and derive identified sets for various parameters under these relaxations. While there are many different ways to relax these assumptions, our goal is to illustrate the breakdown frontier methodology and hence we focus on just one kind of relaxation for each assumption.

Setup

Let $Y_{1}$ and $Y_{0}$ denote the unobserved potential outcomes. Let $X\in\{0,1\}$ be an observed binary treatment. Let $W\in\operatorname*{supp}(W)$ be a vector of observed covariates. This vector may contain discrete covariates, continuous covariates, or both. We observe the scalar outcome variable

[TABLE]

Let $p_{x|w}=\mathbb{P}(X=x\mid W=w)$ denote the observed propensity score. We maintain the following assumption on the joint distribution of $(Y_{1},Y_{0},X,W)$ throughout.

Assumption A3.

For each $x,x^{\prime}\in\{0,1\}$ and $w\in\operatorname*{supp}(W)$:

$Y_{x}\mid X=x^{\prime},W=w$ has a strictly increasing and continuous distribution function on its support, $\operatorname*{supp}(Y_{x}\mid X=x^{\prime},W=w)$. 2. 2.

$\operatorname*{supp}(Y_{x}\mid X=x^{\prime},W=w)=\operatorname*{supp}(Y_{x}\mid W=w)=[\underline{y}_{x}(w),\overline{y}_{x}(w)]$ where $-\infty\leq\underline{y}_{x}(w)<\overline{y}_{x}(w)\leq\infty$. 3. 3.

$p_{x|w}>0$.

Via A1.1, we restrict attention to continuously distributed potential outcomes. A1.2 states that the support of $Y_{x}\mid X=x^{\prime},W=w$ does not depend on $x^{\prime}$, and is a possibly infinite closed interval. This assumption implies that the endpoints $\underline{y}_{x}(w)$ and $\overline{y}_{x}(w)$ are point identified. We maintain A1.2 for simplicity, but it can be relaxed using similar derivations as in Masten and Poirier (2016). A1.3 is an overlap assumption.

Define the conditional rank random variables $U_{0}=F_{Y_{0}\mid W}(Y_{0}\mid W)$ and $U_{1}=F_{Y_{1}\mid W}(Y_{1}\mid W)$. Since $F_{Y_{1}\mid W}(\cdot\mid w)$ and $F_{Y_{0}\mid W}(\cdot\mid w)$ are strictly increasing (by A1.1), $U_{0}\mid W$ and $U_{1}\mid W$ are uniformly distributed on $[0,1]$. The value of unit $i$’s conditional rank random variable $U_{x}$ tells us where unit $i$ lies in the distribution of $Y_{x}\mid W$.

Identifying assumptions

It is well known that the joint conditional distribution of potential outcomes $(Y_{1},Y_{0})\mid W$ is point identified under two assumptions:

Conditional random assignment of treatment: X\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Y_{1}\mid W and X\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Y_{0}\mid W. 2. 2.

Conditional rank invariance: Conditional on $W$, $U_{1}=U_{0}$ almost surely.

Note that the joint conditional independence assumption X\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}(Y_{1},Y_{0})\mid W provides no additional identifying power beyond the marginal conditional independence assumption stated above. Any functional of $F_{Y_{1},Y_{0}\mid W}$ is point identified under these random assignment and rank invariance assumptions. The goal of our identification analysis is to study what can be said about such functionals when one or both of these point-identifying assumptions fails. To do this we define two classes of assumptions: one which indexes the relaxation of random assignment of treatment, and one which indexes the relaxation of rank invariance. These classes of assumptions nest both the point identifying assumptions of random assignment and rank invariance and the opposite end of no constraints on treatment selection or the dependence structure between potential outcomes.

A key feature of the relaxations we use is that they are ‘orthogonal’ in the sense that we can relax each of the two assumptions separately: The amount by which we relax one assumption does not constrain the amount by which we can relax the other assumption. This feature is important since a key goal of our analysis is to quantify the trade-off between relaxations of these two assumptions.

We begin with our measure of distance from conditional independence.

Definition 1**.**

Let $c$ be a scalar between 0 and 1. Say $X$ is conditionally $c$-dependent with $Y_{x}$ given $W$ if

[TABLE]

for $x\in\{0,1\}$ and $w\in\operatorname*{supp}(W)$.

For $c=0$, conditional $c$-dependence implies X\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Y_{1}\mid W and X\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Y_{0}\mid W. For $c>0$, however, it allows for some deviations from conditional independence. Specifically, it allows the unobserved treatment assignment probability $\mathbb{P}(X=1\mid Y_{x}=y,W=w)$ to be at most $c$ probability units away from the observed propensity score $p_{1|w}$. We discuss one way to interpret the magnitude of $c$ on page 4. We give further discussion in our previous paper Masten and Poirier (2018a).

Our second class of assumptions constrains the dependence structure between $Y_{1}$ and $Y_{0}$, conditional on $W$. By Sklar’s Theorem (Sklar 1959), write

[TABLE]

where $C(\cdot,\cdot\mid w)$ is a unique conditional copula function. See Nelsen (2006) for an overview of copulas and Fan and Patton (2014) for a survey of their use in econometrics. Restrictions on $C$ constrain the dependence between potential outcomes. For example, if

[TABLE]

then $U_{1}=U_{0}$ almost surely, conditional on $W$. Thus conditional rank invariance holds. In this case the potential outcomes $Y_{1}$ and $Y_{0}$ are sometimes called conditionally comonotonic and $M$ is called the comonotonicity copula. At the opposite extreme, when $C$ is an arbitrary copula, the dependence between $Y_{1}\mid W$ and $Y_{0}\mid W$ is constrained only by the Fréchet-Hoeffding bounds, which state that

[TABLE]

We next define a class of assumptions which includes both conditional rank invariance and no assumptions on the dependence structure as special cases.

Definition 2**.**

The potential outcomes $(Y_{1},Y_{0})$ satisfy (1$-t)$-percent conditional rank invariance given $W$ if for all $w\in\operatorname*{supp}(W)$ their conditional copula $C$ satisfies

[TABLE]

where $M(u_{1},u_{0})=\min\{u_{1},u_{0}\}$ and $H$ is some conditional copula.

This assumption says that within each covariate cell the population is a mixture of two parts: In one part, rank invariance holds. This part contains $100\cdot t\,$% of the overall population in that cell. In the second part, rank invariance fails in an arbitrary, unknown way. Hence, for this part, the dependence structure is unconstrained beyond the Fréchet-Hoeffding bounds. This part contains $100(1-t)$% of the overall population in that cell. Thus for $t=0$ the usual conditional rank invariance assumption holds, while for $t=1$ no assumptions are made about the dependence structure. For $t\in(0,1)$ we obtain a kind of partial conditional rank invariance. Note that by exercise 2.3 on page 14 of Nelsen (2006), a mixture of copulas like that in equation (3) is also a copula.

To see this mixture interpretation formally, let $T$ follow a Bernoulli distribution with $\mathbb{P}(T=1\mid W=w)=t$, where T\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Y_{1}\mid W and T\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Y_{0}\mid W, but $T$ is not independent of $(Y_{1},Y_{0})\mid W$ jointly. This implies that $T$ has an effect on the dependence structure of $(Y_{1},Y_{0})\mid W$ but not on their conditional marginal distributions. Suppose that individuals for whom $T_{i}=1$ have an arbitrary dependence structure, while those with $T_{i}=0$ have conditionally rank invariant potential outcomes. Then by the law of total probability,

[TABLE]

Our approach to relaxing rank invariance is an example of a more general approach. In this approach we take a weak assumption and a stronger assumption and use them to define a continuous class of assumptions by considering the population as a mixture of two subpopulations. The weak assumption holds in one subpopulation while the stronger assumption holds in the other subpopulation. The mixing proportion $t$ continuously spans the two distinct assumptions we began with. This approach was used earlier by Horowitz and Manski (1995) in their analysis of the contaminated sampling model. While this general approach may not always be the most natural way to relax an assumption, it is always available and hence can be used to facilitate breakdown frontier analyses.

Throughout the rest of this section we impose both conditional $c$-dependence and (1$-t$)-percent conditional rank invariance given $W$.

Assumption A6.

$X$ is conditionally $c$-dependent with the potential outcomes $Y_{x}$ given $W$, where for all $w\in\operatorname*{supp}(W)$ we have $c<\min\{p_{1|w},p_{0|w}\}$. 2. 2.

$(Y_{1},Y_{0})$ satisfies (1$-t$)-percent conditional rank invariance given $W$, where $t\in[0,1]$.

For brevity we focus on the case $c<\min\{p_{1|w},p_{0|w}\}$ throughout this paper. This allows us to explain the key ideas while keeping the notation and derivations relatively simple. All of our results, however, can be relaxed to the general case where $c\in[0,1]$.

Partial identification under relaxations of independence and rank invariance

We next study identification under the deviations from full independence and rank invariance defined above. We begin by briefly recalling results from Masten and Poirier (2018a) on identification of the conditional quantile treatment effect $\text{CQTE}(\tau\mid w)=Q_{Y_{1}\mid W}(\tau\mid w)-Q_{Y_{0}}(\tau\mid w)$, the conditional average treatment effect $\text{CATE}(w)=\mathbb{E}(Y_{1}-Y_{0}\mid W=w)$, and the conditional marginal cdfs of potential outcomes under $c$-dependence. We then derive new identification results for the distribution of treatment effects (DTE), $F_{Y_{1}-Y_{0}}(z)$, and its related parameter $\mathbb{P}(Y_{1}>Y_{0})$.

In Masten and Poirier (2018a), we showed that A1 and A2.1 imply that the identified set for $\text{CQTE}(\tau\mid w)$ is444In that paper we also extended the bounds in equations (4)–(8) to the $c\geq\min\{p_{1|w},p_{0|w}\}$ case. As noted earlier, here we focus on the case $c<\min\{p_{1|w},p_{0|w}\}$ case for brevity.

[TABLE]

where

[TABLE]

We further showed that, assuming $\mathbb{E}(|Y|\mid X=x,W=w)<\infty$ for $x\in\{0,1\}$ and all $w\in\operatorname*{supp}(W)$, the identified set for $\text{CATE}(w)$ is

[TABLE]

and that

[TABLE]

and

[TABLE]

are functionally sharp bounds on the cdf $F_{Y_{x}\mid W}$. We use these conditional cdf bounds in our DTE bounds. Bounds on the corresponding unconditional parameters, like $\text{ATE}=\mathbb{E}[\text{CATE}(W)]$, can be obtained by integrating the conditional bounds over the marginal distribution of $W$. These results are unchanged if we further impose A2.2. That is, assumptions on rank invariance have no identifying power for functionals of the marginal distributions of potential outcomes.

We next derive the identified set for the distribution of treatment effects (DTE), the cdf

[TABLE]

To do this, we first derive the identified set for the conditional distribution of treatment effects (CDTE), the cdf

[TABLE]

By the law of iterated expectations,

[TABLE]

Thus we will obtain the identified set for the DTE by averaging bounds for the CDTE. While the ATE only depends on the conditional marginal distributions of potential outcomes, the CDTE depends on the joint distribution of $(Y_{1},Y_{0})\mid W$. Consequently, as we’ll see below, the identified set for the CDTE depends on the value of $t$.

For any $z\in\mathbb{R}$ define $\mathcal{Y}_{z}(w)=[\underline{y}_{1}(w),\overline{y}_{1}(w)]\cap[\underline{y}_{0}(w)+z,\overline{y}_{0}(w)+z]$. Note that $\operatorname*{supp}(Y_{1}-Y_{0}\mid W=w)\subseteq[\underline{y}_{1}(w)-\overline{y}_{0}(w),\overline{y}_{1}(w)-\underline{y}_{0}(w)]$. Let $z$ be an element of $[\underline{y}_{1}(w)-\overline{y}_{0}(w),\overline{y}_{1}(w)-\underline{y}_{0}(w)]$ such that $\mathcal{Y}_{z}(w)$ is nonempty. If $z$ is such that $\mathcal{Y}_{z}(w)$ is empty, then the CDTE is either 0 or 1 depending solely on the relative location of the two supports, which is point identified by A1.2. In this case, define $\overline{\text{CDTE}}(z,c,t\mid w)$ and $\underline{\text{CDTE}}(z,c,t\mid w)$ to equal this point identified value. If $z>\overline{y}_{1}(w)-\underline{y}_{0}(w)$, define these CDTE bounds to equal 1. If $z<\underline{y}_{1}(w)-\overline{y}_{0}(w)$, define these CDTE bounds to equal 0.

If $\mathcal{Y}_{z}(w)$ is nonempty, define

[TABLE]

[TABLE]

where $U\sim\text{Unif}[0,1]$. The following result shows that (a) these are sharp bounds on the CDTE, and (b) the integral of these bounds over the marginal distribution of $W$ yields sharp bounds on the DTE, defined as $\mathbb{P}(Y_{1}-Y_{0}\leq z)$.

Theorem 1** (DTE bounds).**

Suppose the joint distribution of $(Y,X,W)$ is known. Suppose A1 and A2 hold. Let $z\in\mathbb{R}$. Then the identified set for $\mathbb{P}(Y_{1}-Y_{0}\leq z\mid W=w)$ is

[TABLE]

Moreover, the identified set for $\mathbb{P}(Y_{1}-Y_{0}\leq z)$ is

[TABLE]

The bound functions $\underline{\text{DTE}}(z,\cdot,\cdot)$ and $\overline{\text{DTE}}(z,\cdot,\cdot)$ are continuous and monotonic in both arguments. When both conditional random assignment ($c=0$) and conditional rank invariance ($t=0$) hold, these bounds collapse to a single point and we obtain point identification. If we impose conditional random assignment ($c=0$) but allow arbitrary dependence between $Y_{1}$ and $Y_{0}$ ($t=1$) then we obtain a conditional version of the well known Makarov (1981) bounds. For example, see equation (2) of Fan and Park (2010). DTE bounds have been studied extensively by Fan and coauthors; see the introduction of Fan et al. (2017) for a recent and comprehensive discussion of this literature.

Theorem 1 immediately implies that the identified set for $\mathbb{P}(Y_{1}-Y_{0}>z)$ is

[TABLE]

In particular, setting $z=0$ yields the proportion who benefit from treatment, $\mathbb{P}(Y_{1}>Y_{0})$. Thus theorem 1 allows us to study the sensitivity of this parameter to relaxations of full conditional independence and conditional rank invariance.

Finally, notice that all of the bounds and identified sets discussed in this section are analytically tractable and depend on just three functions identified from the population—the conditional cdf $F_{Y\mid X,W}$, the propensity scores $p_{x|w}$, and the marginal distribution of covariates $F_{W}$. This suggests a plug-in estimation approach which we study in section 3.

Breakdown frontiers

We now formally define the breakdown frontier, which generalizes the scalar breakdown point to multiple assumptions or dimensions. We also define the robust region, the area below the breakdown frontier. These objects can be defined for different conclusions about different parameters in various models. For concreteness, however, we focus on just a few conclusions about $\mathbb{P}(Y_{1}-Y_{0}>z)$ and ATE in the potential outcomes model discussed above.

We begin with the conclusion that $\mathbb{P}(Y_{1}-Y_{0}>z)\geq\underline{p}$ for a fixed $\underline{p}\in[0,1]$ and $z\in\mathbb{R}$. For example, if $z=0$ and $\underline{p}=0.5$, then this conclusion states that at least 50% of people have higher outcomes with treatment than without. If we impose conditional random assignment and conditional rank invariance, then $\mathbb{P}(Y_{1}-Y_{0}>z)$ is point identified and hence we can directly check whether this conclusion holds. But the breakdown frontier approach asks: Relative to these baseline assumptions, what are the weakest assumptions that allow us to draw this conclusion, given the observed distribution of $(Y,X,W)$? Specifically, since larger values of $c$ and $t$ correspond to weaker assumptions, what are the largest values of $c$ and $t$ such that we can still definitively conclude that $\mathbb{P}(Y_{1}-Y_{0}>z)\geq\underline{p}$?

We answer this question in two steps. First we gather all values of $c$ and $t$ such that the conclusion holds. We call this set the robust region. Since the lower bound of the identified set for $\mathbb{P}(Y_{1}-Y_{0}>z)$ is $1-\overline{\text{DTE}}(z,c,t)$ (by theorem 1), the robust region for the conclusion that $\mathbb{P}(Y_{1}-Y_{0}>z)\geq\underline{p}$ is

[TABLE]

The robust region is simply the set of all $(c,t)$ which deliver an identified set for $\mathbb{P}(Y_{1}-Y_{0}>z)$ which lies on or above $\underline{p}$. See pages 60–61 of Stoye (2005) for similar definitions in the scalar assumption case in a different model. Since $\overline{\text{DTE}}(z,c,t)$ is increasing in $c$ and $t$, the robust region will be empty if $\overline{\text{DTE}}(z,0,0)>1-\underline{p}$, and non-empty if $\overline{\text{DTE}}(z,0,0)\leq 1-\underline{p}$. That is, if the conclusion of interest does not hold under the point identifying assumptions, it certainly will not hold under weaker assumptions. From here on we only consider the first case, where the conclusion of interest holds under the point identifying assumptions. That is, we suppose $\overline{\text{DTE}}(z,0,0)\leq 1-\underline{p}$ so that $\text{RR}(z,\underline{p})\neq\emptyset$.

The breakdown frontier is the set of points $(c,t)$ on the boundary of the robust region. Specifically, for the conclusion that $\mathbb{P}(Y_{1}>Y_{0})\geq\underline{p}$, this frontier is the set

[TABLE]

Solving for $t$ in the equation $\overline{\text{DTE}}(0,c,t)=1-\underline{p}$ yields

[TABLE]

where

[TABLE]

and

[TABLE]

Thus we obtain the following analytical expression for the breakdown frontier as a function of $c$:

[TABLE]

This frontier provides the largest relaxations $c$ and $t$ which still allow us to conclude that $\mathbb{P}(Y_{1}>Y_{0})\geq\underline{p}$. It thus provides a quantitative measure of robustness of this conclusion to relaxations of the baseline point identifying assumptions of conditional random assignment and conditional rank invariance. Moreover, the shape of this frontier allows us to understand the trade-off between these two types of relaxations in drawing our conclusion. We illustrate this trade-off between assumptions in our empirical illustration of section 4.

We next consider breakdown frontiers for ATE. Consider the conclusion that $\text{ATE}\geq\mu$ for some $\mu\in\mathbb{R}$. Analogously to above, the robust region for this conclusion is

[TABLE]

and the breakdown frontier is

[TABLE]

These sets are nonempty if $\underline{\text{ATE}}(0)\geq\mu$; that is, if our conclusion holds under the point identifying assumptions. As we mentioned earlier, rank invariance has no identifying power for ATE, and hence the breakdown frontier is a vertical line at the point

[TABLE]

This point $c^{*}$ is a breakdown point for the conclusion that $\text{ATE}\geq\mu$. Note that continuity of $\underline{\text{ATE}}(\cdot)$ implies $\underline{\text{ATE}}(c^{*})=\mu$. Thus we’ve seen two kinds of breakdown frontiers so far: The first had nontrivial curvature, which indicates a trade-off between the two assumptions. The second was vertical in one direction, indicating a lack of identifying power of that assumption.

We can also derive robust regions and breakdown frontiers for more complicated joint conclusions. For example, suppose we are interested in concluding that both $\mathbb{P}(Y_{1}>Y_{0})\geq\underline{p}$ and $\text{ATE}\geq\mu$ hold. Then the robust region for this joint conclusion is just the intersection of the two individual robust regions:

[TABLE]

The breakdown frontier for the joint conclusion is just the boundary of this intersected region. Viewing these frontiers as functions mapping $c$ to $t$, the breakdown frontier for this joint conclusion can be computed as the minimum of the two individual frontier functions. For example, see figure 2c on page 2c.

Above we focused on one-sided conclusions about the parameters of interest. Another natural joint conclusion is the two-sided conclusion that $\mathbb{P}(Y_{1}-Y_{0}>z)\geq\underline{p}$ and $\mathbb{P}(Y_{1}-Y_{0}>z)\leq\overline{p}$, for $0\leq\underline{p}<\overline{p}\leq 1$. No new issues arise here: the robust region for this joint conclusion is still the intersection of the two separate robust regions. Keep in mind, though, that whether we look at a one-sided or a two-sided conclusion is unrelated to the fact that we use lower confidence bands in section 3.

Finally, the bootstrap procedures we propose in section 3 can also be used to do inference on these joint breakdown frontiers. For simplicity, though, in that section we focus on the case where we are only interested in the conclusion $\mathbb{P}(Y_{1}-Y_{0}>z)\geq\underline{p}$.

3 Estimation and inference

In this section we study estimation and inference on the breakdown frontier defined above. The breakdown frontier is a known functional of the conditional cdf of outcomes given treatment and covariates, the probability of treatment given covariates, and the marginal distribution of the covariates. Hence we propose nonparametric sample analog plug-in estimators of the breakdown frontier. We derive $\sqrt{N}$-consistency and asymptotic distributional results using a delta method for directionally differentiable functionals. We then use a bootstrap procedure to construct asymptotically valid lower confidence bands for the breakdown frontier. We conclude by discussing selection of the tuning parameter for this bootstrap procedure.

Although we focus on inference on the breakdown frontier, one might also be interested in doing inference directly on the parameters of interest. If we fix $c$ and $t$ a priori then we obtain identified sets for ATE, QTE, and the DTE from section 2. Our asymptotic results below may be used as inputs to traditional inference on partially identified parameters. See Canay and Shaikh (2017) for a survey of this literature.

To establish our main asymptotic results, we present a sequence of results. Each result focuses on a different component of the overall breakdown frontier: (1) the bounds on the marginal distributions of potential outcomes conditional on $W$, (2) the CQTE bounds, (3) the CATE and ATE bounds, (4) breakdown points for ATE, (5) the CDTE under conditional rank invariance but without full conditional independence, (6) the DTE without either conditional rank invariance or full conditional independence, and finally (7) the breakdown frontier itself.

We first suppose we observe a random sample of data.

Assumption A9.

The random variables $\{(Y_{i},X_{i},W_{i})\}_{i=1}^{N}$ are independently and identically distributed according to the distribution of $(Y,X,W)$.

We assume the support of $W$ is discrete. We sketch an approach to handling continuous covariates in appendix B. Note that $W$ may still be a vector.

Assumption A12.

The support of $W$ is discrete and finite. Let $\operatorname*{supp}(W)=\{w_{1},\ldots,w_{K}\}$.

All parameters of interest are defined as functionals of the underlying parameters $F_{Y\mid X,W}(y\mid x,w)$, $p_{x|w}=\mathbb{P}(X=x\mid W=w)$, and $q_{w}=\mathbb{P}(W=w)$. Let

[TABLE]

[TABLE]

and

[TABLE]

denote the sample analog estimators of these three quantities, which converge uniformly to a Gaussian process at a $\sqrt{N}$-rate; see lemma 5 in appendix C.

Next consider the bounds (7) and (8) on the marginal distributions of potential outcomes. These population bounds are a functional $\phi_{1}$ evaluated at $(F_{Y\mid X,W}(\cdot\mid\cdot,\cdot),p_{(\cdot|\cdot)},q_{(\cdot)})$ where $p_{(\cdot|\cdot)}$ denotes the probability $p_{x|w}$ as a function of $(x,w)\in\{0,1\}\times\operatorname*{supp}(W)$, and $q_{(\cdot)}$ denotes $q_{w}$ as a function of $w\in\operatorname*{supp}(W)$. We estimate these bounds by a plug-in estimator $\phi_{1}(\widehat{F}_{Y\mid X,W}(\cdot\mid\cdot,\cdot),\widehat{p}_{(\cdot|\cdot)},\widehat{q}_{(\cdot)})$. If this functional is differentiable in an appropriate sense, $\sqrt{N}$-convergence in distribution of its arguments will carry over to the functional by the delta method. The type of differentiability we require is Hadamard directional differentiability, first defined by Shapiro (1990) and Dümbgen (1993), and further studied in Fang and Santos (2019).

Definition 3**.**

Let $\mathbb{D}$ and $\mathbb{E}$ be Banach spaces with norms $\|\cdot\|_{\mathbb{D}}$ and $\|\cdot\|_{\mathbb{E}}$. Let $\mathbb{D}_{\phi}\subseteq\mathbb{D}$ and $\mathbb{D}_{0}\subseteq\mathbb{D}$. The map $\phi:\mathbb{D}_{\phi}\rightarrow\mathbb{E}$ is Hadamard directionally differentiable at $\theta\in\mathbb{D}_{\phi}$ tangentially to $\mathbb{D}_{0}$ if there is a continuous map $\phi^{\prime}_{\theta}:\mathbb{D}_{0}\rightarrow\mathbb{E}$ such that

[TABLE]

for all sequences $\{h_{n}\}\subset\mathbb{D}$ and $\{t_{n}\}\in\mathbb{R}_{+}$ such that $t_{n}\searrow 0$, $\|h_{n}-h\|_{\mathbb{D}}\rightarrow 0$, $h\in\mathbb{D}_{0}$ as $n\rightarrow\infty$, and $\theta+t_{n}h_{n}\in\mathbb{D}_{\phi}$ for all $n$.

If we further have that $\phi_{\theta}^{\prime}$ is linear, then we say $\phi$ is Hadamard differentiable (proposition 2.1 of Fang and Santos 2019). Not every Hadamard directional derivative $\phi_{\theta}^{\prime}$ must be linear, however.

We use the functional delta method for Hadamard directionally differentiable mappings (e.g., theorem 2.1 in Fang and Santos 2019) to show convergence in distribution of our estimators. Such convergence is usually to a non-Gaussian limiting process. We do not use this distribution to do inference since obtaining analytical asymptotic confidence bands would be challenging. Instead, we use a bootstrap procedure to obtain asymptotically valid uniform confidence bands for our breakdown frontier and associated estimators.

Returning to our population bounds (7) and (8), we estimate these by

[TABLE]

Note that these estimators may not perform well when $c$ is close to $p_{x|w}$. In our analysis we assume $c$ is bounded away from $p_{x|w}$.

In addition to assumption A1, we make the following regularity assumptions.

Assumption A15.

For each $x\in\{0,1\}$ and $w\in\operatorname*{supp}(W)$, $-\infty<\underline{y}_{x}(w)<\overline{y}_{x}(w)<+\infty$. 2. 2.

For each $x\in\{0,1\}$ and $w\in\operatorname*{supp}(W)$, $F_{Y\mid X,W}(y\mid x,w)$ is continuously differentiable everywhere with density $f_{Y\mid X,W}(y\mid x,w)$ uniformly continuous in $y$, uniformly bounded from above, and uniformly bounded away from zero on $\operatorname*{supp}(Y\mid X=x,W=w)$.

A5.1 combined with our earlier assumption A1.2 constrain the potential outcomes to have compact support. This compact support assumption is not used to analyze our cdf bounds estimators (10), but we use it later to obtain estimates of the corresponding quantile function bounds uniformly over their arguments $u\in(0,1)$, which we then use to estimate the bounds on $\mathbb{P}(Q_{Y_{1}\mid W}(U\mid w)-Q_{Y_{0}\mid W}(U\mid w)\leq z)$. This is a well known issue when estimating quantile processes; for example, see van der Vaart (2000) lemma 21.4(ii). A5.2 requires the density of $Y\mid X,W$ to be bounded away from zero uniformly. This ensures that conditional quantiles of $Y\mid X,W$ are uniquely defined. It also implies that the limiting distribution of the estimated quantile bounds will be well-behaved. Uniform continuity of the density implies that the derivatives of the conditional quantile function with respect to $\tau$ are uniformly continuous.

For some of our main results in this section (lemmas 1, 3, 4 and theorem 2) we establish convergence uniformly over $c\in\mathcal{C}$ for some finite grid $\mathcal{C}=\{c_{1},c_{2},\ldots,c_{J}\}\subset[0,\min\{p_{1|w},p_{0|w}\})$. We discuss the choice of these grid points on page 3. We constrain this grid to be below $\min\{p_{1|w},p_{0|w}\}$ solely for simplicity, as all our results can be extended to grids $\mathcal{C}\subset[0,1]$ by combining our present bound estimates with estimates based on the $c\geq\min\{p_{1|w},p_{0|w}\}$ case given in Masten and Poirier (2018a). Weak convergence does not hold uniformly over an interval of $c$ since some of the functionals below are not Hadamard directionally differentiable when their codomain is a set of functions on that interval. To resolve this issue, we propose two ways of conducting inference on the breakdown frontier uniformly over intervals of $c$. The first is to use the fixed grid and monotonicity of the breakdown frontier to construct a uniform band. The second is to smooth the population breakdown frontier such that it is Hadamard differentiable when viewed as a function of $c$. We use the first approach in this section and the second approach in supplemental appendix D.

The next result establishes convergence in distribution of the cdf bound estimators. Here and below we use the following notation: For an arbitrary set $\mathcal{A}$ and a Banach space $\mathcal{B}$, $\ell^{\infty}(\mathcal{A},\mathcal{B})$ denotes the set of all maps $z:\mathcal{A}\rightarrow\mathcal{B}$ with finite sup-norm $\|z\|=\sup_{a\in\mathcal{A}}\|z(a)\|_{\mathcal{B}}$, equipped with this norm. For example, see van der Vaart and Wellner (1996, page 381).

Lemma 1**.**

Suppose A1, A3, and A4 hold. Let $\mathcal{Y}\subset\mathbb{R}$ be a finite grid of points. Then

[TABLE]

a tight random element of $\ell^{\infty}(\mathcal{Y}\times\{0,1\}\times\operatorname*{supp}(W)\times\mathcal{C},\mathbb{R}^{2})$.

$\mathbf{Z}_{2}$ is not Gaussian itself, but it is a continuous transformation of Gaussian processes. For given $(x,c,w)$, the limit will be Gaussian at all values of $y$ except for

[TABLE]

A characterization of this limiting process is given in the proof in appendix C.

Next consider the conditional quantile bounds (5), which we estimate by

[TABLE]

The next result establishes uniform convergence in distribution of these quantile bounds estimators. For the following results, let $\overline{C}\in(0,\min\{p_{1|w},p_{0|w}\})$ for all $w\in\operatorname*{supp}(W)$.

Lemma 2**.**

Suppose A1, A3, A4, and A5 hold. Then

[TABLE]

a mean-zero Gaussian process in $\ell^{\infty}((0,1)\times\{0,1\}\times\operatorname*{supp}(W)\times[0,\overline{C}],\mathbb{R}^{2})$ with continuous paths.

This result is uniform in $c$ on an interval, in $x\in\{0,1\}$, $w\in\operatorname*{supp}(W)$, and in $\tau\in(0,1)$. This result directly implies convergence over $c\in\mathcal{C}$ as well. Unlike the distribution of the cdf bounds estimators, this process is Gaussian. This follows by Hadamard differentiability of the mapping between $\theta_{0}\equiv(F_{Y\mid X,W}(\cdot\mid\cdot,\cdot),p_{(\cdot|\cdot)},q_{(\cdot)})$ and the conditional quantile bounds. By applying the functional delta method, we can show asymptotic normality of smooth functionals of these bounds. A first set of functionals are the CQTE bounds of equation (4), which are a linear combination of the quantile bounds. Let

[TABLE]

and

[TABLE]

Then,

[TABLE]

where the superscript $\mathbf{Z}^{(j)}$ denotes the $j$th component of the vector $\mathbf{Z}$.

A second set of functionals are the CATE bounds from equation (6). These bounds are smooth linear functionals of the CQTE bounds. Therefore the joint asymptotic distribution of these bounds can be established by the continuous mapping theorem. Let

[TABLE]

Then, by the linearity of the integral operator, these estimated CATE bounds converge to their population counterpart at a $\sqrt{N}$-rate and therefore

[TABLE]

a mean-zero Gaussian process in $\ell^{\infty}(\operatorname*{supp}(W)\times[0,\overline{C}],\mathbb{R}^{2})$ with continuous paths.

We estimate the unconditional ATE bounds by integrating over the empirical distribution of the covariates $W$: Let

[TABLE]

The following decomposition implies that the estimated ATE upper bound converges weakly to a Gaussian element:

[TABLE]

A similar result holds for the estimated ATE lower bound.

Next consider estimation of the breakdown point for the claim that $\text{ATE}\geq\mu$ where $\mu\in\mathbb{R}$. To focus on the nondegenerate case, suppose the population value of ATE obtained under full independence is greater than $\mu$, $\underline{\text{ATE}}(0)>\mu$ which implies $c^{*}>0$. Let

[TABLE]

be the estimated breakdown point. This is the estimated smallest relaxation of independence such that we cannot conclude that the ATE is strictly greater than $\mu$. By the properties of the quantile bounds as a function of $c$, the function $\underline{\text{ATE}}(c)$ is non-increasing and differentiable in $c$. We now present a result about the asymptotic distribution of $\widehat{c}^{*}$.

Proposition 1**.**

Suppose A1, A3, A4, and A5 hold. Assume $c^{*}\in(0,\overline{C}]$. Then $\sqrt{N}(\widehat{c}^{*}-c^{*})\rightsquigarrow\mathbf{Z}_{4}$, a Gaussian random variable.

The assumption that $c^{*}\in(0,\overline{C}]$ can again be relaxed to the general case where $c^{*}\in(0,1]$ but we maintain the stronger assumption for brevity.

Under conditional rank invariance, we can also establish asymptotic normality of bounds for $\mathbb{P}(Q_{Y_{1}\mid W}(U\mid w)-Q_{Y_{0}\mid W}(U\mid w)\leq z)$ for a fixed $z\in\mathbb{R}$. These bounds are given by

[TABLE]

We keep $z$ implicit in the notation for these bounds. Estimates for these quantities are provided by

[TABLE]

Asymptotic normality can be established using the Hadamard directional differentiability of the mapping from the differences in quantile bounds to the bounds $(\underline{P}(c\mid w),\overline{P}(c\mid w))$. This mapping is called the pre-rearrangement operator. Chernozhukov et al. (2010) showed that this operator was Hadamard differentiable when the quantile functions are continuously differentiable for all $u\in(0,1)$. In our case, the underlying quantile functions are continuously differentiable on $(0,1/2)\cup(1/2,1)$, and continuous but not differentiable at $u=1/2$. At this value, the left and right derivatives exist and are finite, but are generally different from one another. We extend the result of Chernozhukov et al. (2010) to the case where the quantile function has a point of non-differentiability by showing Hadamard directional differentiability of this mapping.

To do so, we make additional assumptions on the behavior of these quantile functions.

Assumption A18.

For each $c\in\mathcal{C}$ and $w\in\operatorname*{supp}(W)$,

The number of elements in each of the sets

[TABLE]

is finite. 2. 2.

The following hold.

1. (a)

For any $u\in\mathcal{U}^{*}_{1}(c\mid w)$, $\overline{Q}^{c}_{Y_{1}\mid W}(u\mid w)-\underline{Q}^{c}_{Y_{0}\mid W}(u\mid w)\neq z$. 2. (b)

For any $u\in\mathcal{U}^{*}_{2}(c\mid w)$, $\underline{Q}^{c}_{Y_{1}\mid W}(u\mid w)-\overline{Q}^{c}_{Y_{0}\mid W}(u\mid w)\neq z$.

These assumptions imply that the respective function’s derivatives change signs a finite number of times, and therefore they cross the horizontal line at $z$ a finite number of times. These functions are continuously differentiable in $u$ everywhere on $(0,1/2)\cup(1/2,1)$, and are directionally differentiable at $1/2$. The second assumption rules out the functions being flat when exactly valued at $z$. Failure of the second condition in this assumption implies that convergence will hold uniformly over any compact subset that excludes these values, which typically form a measure-zero set. Therefore this assumption can be satisfied by considering convergence for values of $c$ which exclude those where the second part of assumption A6 fails. Without knowing a priori at which values this assumption may fail, selecting grid points randomly from a continuous distribution ensures that these values are selected with probability zero.

An alternative approach to inference if the second condition fails for some values of $c$ is to smooth the population function using methods described in supplemental appendix D. Like in Chernozhukov et al. (2010, corollary 4), we require a tuning parameters to control the level of smoothing. We show that $\sqrt{N}$-convergence holds for all parameter values when introducing any amount of fixed smoothing.

Finally, note that A6 is refutable, since it is expressed as a function of identified quantities, namely the CQTE bounds for all $u\in(0,1)$.

With this additional assumption we can show $\sqrt{N}$-convergence of the bounds uniformly in $\operatorname*{supp}(W)\times\mathcal{C}$.

Lemma 3**.**

Suppose A1, A3, A4, A5, and A6 hold. Then

[TABLE]

a tight random element in $\ell^{\infty}(\operatorname*{supp}(W)\times\mathcal{C},\mathbb{R}^{2})$.

If conditional random assignment holds ($c=0$) in addition to conditional rank invariance ($t=0$), then the CDTE is point identified and lemma 3 gives the asymptotic distribution of the sample analog CDTE estimator (in this case the upper and lower bound functions are equal). This can be considered an estimator of the CDTE in one of the models of Matzkin (2003).

We now establish the limiting distribution of the CDTE bounds uniformly in $(c,t,w)$. Let

[TABLE]

We estimate the unconditional DTE bounds by integrating the estimated CDTE bounds over the empirical distribution of the covariates: Let

[TABLE]

We have shown in lemma 3 that the terms $\underline{P}(c\mid w)$ and $\overline{P}(c\mid w)$ are estimated at a $\sqrt{N}$-rate by the Hadamard directional differentiability of the mapping linking empirical cdfs and these terms. We now show that the second components of the CDTE bounds are a Hadamard directionally differentiable functional as well, leading to the $\sqrt{N}$ joint convergence of the DTE bounds to a tight, random element uniformly in $c$ and $t$.

Lemma 4**.**

Fix $z\in\mathbb{R}$. Suppose A1, A3, A4, A5, and A6 hold. Then

[TABLE]

a tight random element of $\ell^{\infty}(\mathcal{C}\times[0,1],\mathbb{R}^{2})$ with continuous paths.

Having established the convergence in distribution of the DTE, we can now show that the breakdown frontier also converges in distribution uniformly over its arguments. Denote the estimated breakdown frontier for the conclusion that $\mathbb{P}(Y_{1}>Y_{0})\geq\underline{p}$ by

[TABLE]

where

[TABLE]

with

[TABLE]

By combining our previous lemmas, we can show that the estimated breakdown frontier converges in distribution.

Theorem 2**.**

Suppose A1, A3, A4, A5, and A6 hold. Let $\mathcal{P}\subset[0,1]$ be a finite grid of points. Then

[TABLE]

a tight random element of $\ell^{\infty}(\mathcal{C}\times\mathcal{P})$.

This result essentially follows from the convergence of the preliminary estimators established in lemma 5 in appendix C and by showing that the breakdown frontier is a composition of a number of Hadamard differentiable and Hadamard directionally differentiable mappings, implying convergence in distribution of the estimated breakdown frontier.

Breakdown frontiers for more complex conclusions can typically be constructed from breakdown frontiers for simpler conclusions. For example, consider the breakdown frontier for the joint conclusion that $\mathbb{P}(Y_{1}>Y_{0})\geq\underline{p}$ and $\text{ATE}\geq\mu$. Then the breakdown frontier for this joint conclusion is the minimum of the two individual frontier functions. Alternatively, consider the conclusion that $\mathbb{P}(Y_{1}>Y_{0})\geq\underline{p}$ or $\text{ATE}\geq\mu$, or both, hold. Then the breakdown frontier for this joint conclusion is the maximum of the two individual frontier functions. Since the minimum and maximum operators are Hadamard directionally differentiable, these joint breakdown frontiers will also converge in distribution.

Since the limiting process is non-Gaussian, inference on the breakdown frontier is not based on standard errors as with Gaussian limiting theory. Our processes’ distribution is characterized fully by the expressions in appendix C, but obtaining analytical estimates of quantiles of functionals of these processes would be challenging. In the next subsection we give details on the bootstrap procedure we use to construct confidence bands for the breakdown frontier.

Bootstrap inference

As mentioned earlier, we use a bootstrap procedure to do inference on the breakdown frontier rather than directly using its limiting process. In this subsection we discuss how to use the bootstrap to approximate this limiting process. In the next subsection we discuss its application to constructing uniform confidence bands.

First we define some general notation. Let $Z_{i}=(Y_{i},X_{i},W_{i})$ and $Z^{N}=\{Z_{1},\ldots,Z_{N}\}$. Let $\theta_{0}$ denote some parameter of interest and let $\widehat{\theta}$ be an estimator of $\theta_{0}$ based on the data $Z^{N}$. Let $\mathbf{A}_{N}^{*}$ denote $\sqrt{N}(\widehat{\theta}^{*}-\widehat{\theta})$ where $\widehat{\theta}^{*}$ is a draw from the nonparametric bootstrap distribution of $\widehat{\theta}$. Suppose $\mathbf{A}$ is the tight limiting process of $\sqrt{N}(\widehat{\theta}-\theta_{0})$. Denote bootstrap consistency by $\mathbf{A}_{N}^{*}\overset{P}{\rightsquigarrow}\mathbf{A}$ where $\overset{P}{\rightsquigarrow}$ denotes weak convergence in probability, conditional on the data $Z^{N}$. Weak convergence in probability conditional on $Z^{N}$ is defined as

[TABLE]

where $\text{BL}_{1}$ denotes the set of Lipschitz functions into $\mathbb{R}$ with Lipschitz constant no greater than 1.

We focus on the following specific choices of $\theta_{0}$ and $\widehat{\theta}$:

[TABLE]

For these choices, let $\mathbf{Z}_{N}^{*}=\sqrt{N}(\widehat{\theta}^{*}-\widehat{\theta})$. Let $\mathbf{Z}_{1}$ denote the limiting distribution of $\sqrt{N}(\widehat{\theta}-\theta_{0})$; see lemma 5 in appendix C. Theorem 3.6.1 of van der Vaart and Wellner (1996) implies that $\mathbf{Z}_{N}^{*}\overset{P}{\rightsquigarrow}\mathbf{Z}_{1}$. Our parameters of interest are all functionals $\phi$ of $\theta_{0}$. For Hadamard differentiable functionals $\phi$, the nonparametric bootstrap is consistent. For example, see theorem 3.1 of Fang and Santos (2019). They further show that $\phi$ is Hadamard differentiable if and only if

[TABLE]

where $\phi_{\theta_{0}}^{\prime}$ denotes the Hadamard derivative at $\theta_{0}$. This implies that the nonparametric bootstrap can be used to do inference on the QTE and ATE bounds since they are Hadamard differentiable functionals of $\theta_{0}$. A second implication is that the nonparametric bootstrap is not consistent for the DTE or for the breakdown frontier for claims about the DTE since they are Hadamard directionally differentiable mappings of $\theta_{0}$, but they are not ordinary Hadamard differentiable.

In such cases, Fang and Santos (2019) show that a different bootstrap procedure is consistent. Specifically, let $\widehat{\phi}_{\theta_{0}}^{\prime}$ be a consistent estimator of $\phi^{\prime}_{\theta_{0}}$. Then their results imply that

[TABLE]

Analytical consistent estimates of $\phi^{\prime}_{\theta_{0}}$ are often difficult to obtain, so Dümbgen (1993) and Hong and Li (2018) propose using a numerical derivative estimate of $\phi^{\prime}_{\theta_{0}}$. Their estimate of the limiting distribution of $\sqrt{N}(\phi(\widehat{\theta})-\phi(\theta_{0}))$ is given by the distribution of

[TABLE]

across the bootstrap estimates $\widehat{\theta}^{*}$. Under the rate constraints $\varepsilon_{N}\rightarrow 0$ and $\sqrt{N}\varepsilon_{N}\rightarrow\infty$, and some measurability conditions stated in their appendix, Hong and Li (2018) show

[TABLE]

where the left hand side is defined in equation (14).

This bootstrap procedure requires evaluating $\phi$ at two values, which is computationally simple. It also requires selecting the tuning parameter $\varepsilon_{N}$, which we discuss later. Note that the standard, or naive, bootstrap is a special case of this numerical delta method bootstrap where $\varepsilon_{N}=N^{-1/2}$.

Uniform confidence bands for the breakdown frontier

In this subsection we combine all of our asymptotic results thus far to construct uniform confidence bands for the breakdown frontier. As in section 2 we use the function $\text{BF}(\cdot,\underline{p})$ to characterize this frontier. We specifically construct one-sided lower uniform confidence bands. That is, we will construct a lower band function $\widehat{\text{LB}}(c)$ such that

[TABLE]

We use a one-sided lower uniform confidence band because this gives us an inner confidence set for the robust region. Specifically, define the set

[TABLE]

Then validity of the confidence band $\widehat{\text{LB}}$ implies

[TABLE]

Thus the area underneath our confidence band, $\text{RR}_{L}$, is interpreted as follows: Across repeated samples, approximately $100(1-\alpha)$% of the time, every pair $(c,t)\in\text{RR}_{L}$ leads to a population level identified set for the parameter $\mathbb{P}(Y_{1}>Y_{0})$ which lies weakly above $\underline{p}$. Put differently, approximately $100(1-\alpha)$% of the time, every pair $(c,t)\in\text{RR}_{L}$ still lets us draw the conclusion we want at the population level. Hence the size of this set $\text{RR}_{L}$ is a finite sample measure of robustness of our conclusion to failure of the point identifying assumptions. We discuss an alternative testing-based interpretation on page 5 in supplemental appendix A.

One might be interested in constructing one-sided upper confidence bands if the goal was to do inference on the set of assumptions for which we cannot come to the conclusion of interest. This might be useful in situations where two opposing sides are debating a conclusion. But since our focus is on trying to determine when we can come to the desired conclusion, rather than looking for when we cannot, we only describe the one-sided lower confidence band case.

When studying inference on scalar breakdown points, Kline and Santos (2013) constructed one-sided lower confidence intervals. Unlike for breakdown frontiers, uniformity over different points in the assumption space is not a concern for inference on breakdown points. See supplemental appendix A for more discussion.

We consider bands of the form

[TABLE]

for some function $\widehat{k}(\cdot)\geq 0$. This band is an asymptotically valid lower uniform confidence band of level $1-\alpha$ if

[TABLE]

or, equivalently, if

[TABLE]

In our theoretical analysis, we consider $\widehat{k}(c)=\widehat{z}_{1-\alpha}\sigma(c)$ for a scalar $\widehat{z}_{1-\alpha}$ and a function $\sigma$. We focus on known $\sigma$ for simplicity. We start by deriving a uniform band over a grid $\mathcal{C}$, then extend it over an interval using monotonicity of the breakdown frontier. As discussed earlier, we only derive uniformity of the band over $c\in[0,\overline{C}]$ rather than over $c\in[0,1]$, but this is also for brevity and can be relaxed. The choice of $\sigma$ affects the shape of the confidence band, and there are many possible choices of the function $\sigma$ which yield valid level $1-\alpha$ uniform confidence bands. See Freyberger and Rai (2018) for a detailed analysis. A simple choice of $\sigma$ is the constant function: $\sigma(c)=1$, which delivers an equal width uniform band. Alternatively, as we do below, one could choose $\sigma(c)$ to construct a minimum width confidence band (equivalently, maximum area of $\text{RR}_{L}$).

Proposition 2**.**

Suppose A1, A3, A5, and A6 hold. Define $\phi:\ell^{\infty}(\mathbb{R}\times\{0,1\},\mathbb{R}^{2})\rightarrow\ell^{\infty}(\mathcal{C})$ such that

[TABLE]

Then $\phi$ is Hadamard directionally differentiable. Suppose that $\varepsilon_{N}\rightarrow 0$ and $\sqrt{N}\varepsilon_{N}\rightarrow\infty$. Let $\widehat{\theta}^{*}$ denote a draw from the nonparametric bootstrap distribution of $\widehat{\theta}$. Then

[TABLE]

For a given function $\sigma(\cdot)$ such that $\inf_{c\in\mathcal{C}}\sigma(c)>0$, define

[TABLE]

Finally, suppose also that the cdf of

[TABLE]

is continuous and strictly increasing at its $1-\alpha$ quantile, denoted $z_{1-\alpha}$. Then $\widehat{z}_{1-\alpha}=z_{1-\alpha}+o_{p}(1)$.

This proposition is a variation of corollary 3.2 in Fang and Santos (2015). As a consequence of this result, the lower $1-\alpha$ band $\widehat{\text{LB}}(c)=\widehat{\text{BF}}(c,\underline{p})-\widehat{z}_{1-\alpha}\sigma(c)$ is valid uniformly on the grid $\mathcal{C}$. To extend the uniformity to all of $[0,\overline{C}]$ we propose the following lower confidence band:

[TABLE]

This band is a step function which interpolates between grid points using the least monotone interpolation. The following result shows its validity.

Corollary 1**.**

Let the assumptions of proposition 2 hold. Then, $\widetilde{\text{LB}}(c)$ is a uniform lower $1-\alpha$ band for $\text{BF}(c,\underline{p})$ over $c\in[0,\overline{C}]$.

Corollary 1 shows that, for any fixed $J\geq 1$, the interpolated lower confidence band preserves exact the $1-\alpha$ coverage on the grid points. This follows by monotonicity of the breakdown frontier; see lemma 10 in appendix C. That said, this interpolated band might not be taut, in the sense that there may exist other lower bands with $1-\alpha$ coverage that are weakly larger than $\widetilde{\text{LB}}(c)$ for all $c$ and strictly larger at some values of $c$. See Freyberger and Rai (2018) for further discussion of taut confidence bands.

Proposition 2 can be extended to estimated functions $\sigma$, although we leave the details for future work. We use an estimated $\sigma$ in our application, as described next. When both $z_{1-\alpha}$ and $\sigma$ are estimated, we work directly with $\widehat{k}(c)=\widehat{z}_{1-\alpha}\widehat{\sigma}(c)$. We choose $\widehat{k}(c)$ to minimize an approximation to the area between the confidence band and the estimated function; equivalently, to maximize the area of $\text{RR}_{L}$. Specifically, we let $\widehat{k}(c_{1}),\ldots,\widehat{k}(c_{J})$ solve

[TABLE]

subject to

[TABLE]

where we approximate the left-hand side probability via the numerical delta method bootstrap. The criterion function here is just a right Riemann sum over the grid points. This optimization is not computationally costly: It is only performed once per value of $\underline{p}$ and $\varepsilon_{N}$. Moreover, in our empirical illustration it takes an average of 15 seconds per run on a mid-2013 MacBook Air.

Choosing the grid points $\mathcal{C}$

Here we suggest three approaches for choosing the number and location of the grid points $\mathcal{C}=\{c_{1},\ldots,c_{J}\}$. First, one can let $\mathcal{C}=\{\overline{c}_{1},\ldots,\overline{c}_{K}\}$ where $K$ is the number of observed covariates and, for each $k\in\{1,\ldots,K\}$, $\overline{c}_{k}$ is the maximal deviation between the observed propensity score and the “leave out variable $k$” propensity score, which we define and discuss in our empirical illustration on page 4. There we argue that these are natural points to consider.

Second, researchers can choose equally spaced grid points, for a fixed $J$. Third, researchers can randomly select grid points from a continuous distribution, for a fixed $J$. As mentioned on page 6, this random selection ensures that assumption A6 holds with probability one. Both of these approaches are commonly used in the literature. Moreover, both of these approaches can be used in combination with the first approach. In this case, researchers may want to begin with $\{\overline{c}_{1},\ldots,\overline{c}_{K}\}$ and then add a multiple $m$ of $K$ additional points, so that the total number of points $J$ is $K+mK$ for some positive integer $m$.

Finally, we emphasize that for our asymptotic approximations to be valid, the key restriction is that the grid $\mathcal{C}$ is not too dense around the points $c$ of nondifferentiability. Picking a sufficiently sparse fixed finite grid is one way to ensure this. Thus, although we state our formal results for a fixed grid, one could instead let $J\rightarrow\infty$ sufficiently slowly as $N\rightarrow\infty$. Alternatively, one could pre-estimate the points of nondifferentiability and then let $\mathcal{C}$ equal the entire domain, minus sufficiently large intervals around these points. The length of these removed intervals then shrinks asymptotically. Horowitz and Lee (2012, 2017) discuss approaches like these in different settings.

Bootstrap selection of $\varepsilon_{N}$

While Dümbgen (1993) and Hong and Li (2018) provide rate constraints on $\varepsilon_{N}$, they do not recommend a procedure for picking $\varepsilon_{N}$ in practice. In this section, we suggest a heuristic bootstrap method for picking $\varepsilon_{N}$. We use this method for our empirical illustration in section 4; we also present the full range of bands considered. Since the question of choosing $\varepsilon_{N}$ goes beyond the purpose of the present paper, we defer a formal analysis of this method to future research. For discussions of bootstrap selection of tuning parameters in other problems, see Taylor (1989), Léger and Romano (1990), Marron (1992), and Cao et al. (1994).

Fix a $\underline{p}$. Let $\text{CP}_{N}(\varepsilon;F_{Y,X,W})$ denote the finite sample coverage probability of our confidence band as described above, for a fixed $\varepsilon$. This statistic depends on the unknown distribution of the data, $F_{Y,X,W}$. The bootstrap replaces $F_{Y,X,W}$ with an estimator $\widehat{F}_{Y,X,W}$. We pick a grid $\{\varepsilon_{1},\ldots,\varepsilon_{K}\}$ of $\varepsilon$’s and let $\widehat{\varepsilon}_{N}$ solve

[TABLE]

We compute $\text{CP}_{N}$ by simulation. In our empirical illustration, we take $B=500$ draws. We use the same grid of $\varepsilon$’s as in our Monte Carlo simulations in supplemental appendix C. Larger grids and larger values of $B$ can be chosen subject to computational constraints. We furthermore must choose an estimator $\widehat{F}_{Y,X,W}$. The nonparametric bootstrap uses the empirical distribution. We use the smoothed bootstrap (De Angelis and Young 1992, Polansky and Schucany 1997). Specifically, we estimate the distribution of $(X,W)$ by its empirical distribution. We then let $\widehat{F}_{Y\mid X,W}$ be a kernel smoothed cdf estimate of the conditional cdf of $Y|X,W$. We use the standard logistic cdf kernel and the method proposed by Hansen (2004) to choose the smoothing bandwidths. We divide these bandwidths in half since this visually appears to better capture the shape of the conditional empirical cdfs, and since smaller order bandwidths are recommended for the smoothed bootstrap (section 4 of De Angelis and Young 1992).

Bootstrap consistency requires sufficient smoothness of the functional of interest in the underlying cdf. It may be that the lack of smoothness that requires us to use the methods of Fang and Santos (2019) and Hong and Li (2018) in the first place also cause the naive bootstrap to be inconsistent for approximating the distribution of $\text{CP}_{N}(\varepsilon;F_{Y,X,W})$. As mentioned earlier, formally investigating this issue is beyond the scope of this paper. Our goal here is merely to suggest a simple first-pass approach at choosing $\varepsilon_{N}$.

4 Empirical illustration: The effects of child soldiering

In this section we use our results to examine the impact of assumptions in determining the effects of child soldiering on wages. We first briefly discuss the background and then we present our analysis.

Background

We use data from phase 1 of SWAY, the Survey of War Affected Youth in northern Uganda, conducted by principal researchers Jeannie Annan and Chris Blattman (see Annan et al. 2006). As Blattman and Annan (2010) discuss on page 882, a primary goal of this survey was to understand the effects of a twenty year war in Uganda, where “an unpopular rebel group has forcibly recruited tens of thousands of youth”. In that paper, they use this data to examine the impacts of abduction on educational, labor market, psychosocial, and health outcomes. In our illustration, we focus solely on the impact of abduction on wages.

Blattman and Annan note that self-selection into the military is a common problem in the literature studying the effects of military service on outcomes. They argue that forced recruitment in Uganda led to random assignment of military service in their data. They first provide qualitative evidence for this, based on interviews with former rebels who led raiding parties. After murdering and mutilating civilians, the rebels had no public support, making abduction the only means of recruitment. Youths were generally taken during nighttime raids on rural households. According to the former rebel leaders, “targets were generally unplanned and arbitrary; they raided whatever homesteads they encountered, regardless of wealth or other traits.”

This qualitative evidence is supported by their survey data, where Blattman and Annan show that most pre-treatment covariates are balanced across the abducted and nonabducted groups (see their table 2). Only two covariates are not balanced: year of birth and prewar household size. They say this is unsurprising because

“a youth’s probability of ever being abducted depended on how many years of the conflict he fell within the [rebel group’s] target age range. Moreover, abduction levels varied over the course of the war, so youth of some ages were more vulnerable to abduction than others. The significance of household size, meanwhile, is driven by households greater than 25 in number. We believe that rebel raiders, who traveled in small bands, were less likely to raid large, difficult-to-control households.” (Page 887)

Hence they use a selection-on-observables identification strategy, conditioning on these two variables.

While their evidence supporting the full conditional independence assumption is compelling, this assumption is still nonrefutable. Hence they apply the methods of Imbens (2003) to analyze the sensitivity to this assumption. In this analysis they only consider one outcome variable, years of education. Likewise, as in Imbens (2003), they only look at one parameter, the constant treatment effect in a fully parametric model.

We complement their results by applying the breakdown frontier methods we develop in this paper. We focus on the log-wage outcome variable. We look at both the average treatment effect and $\mathbb{P}(Y_{1}>Y_{0})$, which was not studied in Blattman and Annan (2010).

Analysis

The original phase 1 SWAY data has 1216 males born between 1975 and 1991. Of these, wage data is available for 504 observations. 56 of these earned zero wages; we drop these and only look at people who earned positive wages. This leaves us with our main sample of 448 observations. In addition to this outcome variable, we let our treatment variable be an indicator that the person was not abducted. We include the two covariates discussed above, age when surveyed and household size in 1996. Additional covariates can be included, but we focus on just these two for simplicity.

Table 1 shows summary statistics for these four variables. 36% of our sample were not abducted. Age ranges from 14 years old to 30 years old, with a median of 22 years old. Household size ranges from 2 people to 28, with a median of 8 people. Wages range from as low as 36 shillings to as high as about 83,300 shillings, with a median of 1,400 shillings.

Age has 17 support points and household size has 21 support points. Hence there are 357 total covariate cells. Including the treatment variable, this yields 714 total cells, compared to our sample size of 448 observations. Since we focus on unconditional parameters, having small or zero observations per cell is not a problem in principle. However, in the finite sample we have, to ensure that our estimates of the cdf bounds $\overline{F}_{Y_{x}\mid W}^{c}(y\mid w)$ and $\underline{F}_{Y_{x}\mid W}^{c}(y\mid w)$ are reasonably smooth in $y$, we collapse our covariates as follows. We replace age with a binary indicator of whether one is above or below the median age. Likewise, we replace household size with a binary indicator of whether one lived in a household with above or below median household size. This reduces the number of covariate cells to 4, giving 8 total cells including the treatment variable. This yields approximately 55 observations per cell. While this crude approach suffices for our illustration, in more extensive empirical analyses one may want to use more sophisticated methods. For example, we could use discrete kernel smoothing, as discussed in Li and Racine (2008), who also provide additional references. We also consider alternative coarsenings in supplemental appendix E.

Table 2 shows unconditional comparisons of means of the outcome and the original covariates across the treatment and control groups. Wages for people who were not abducted are 702 shillings larger on average. People who were not abducted are also about 1.4 years younger than those who were abducted. People who were not abducted also had a slightly larger household size than those who were abducted. Only the difference in ages is statistically significant at the usual levels, but as in tables 2 and 3 of Blattman and Annan (2010) the standard errors can be decreased by including additional controls. These extra covariates are not essential for illustrating our breakdown frontier methods, however.

The point estimates in table 2 are unconditional on the two covariates. Next we consider the conditional independence assumption, with age and household size in 1996 as our covariates. Under this assumption, our estimate of ATE is 890 [726.13] shillings when the outcome variable is level of wages, and is 0.21 [0.11] when the outcome variable is log wage.555Using their full set of control variables, Blattman and Annan (2010) estimate ATE to be 0.33 [0.15] when the outcome is log wage. See column 1 of their table 3. To check the robustness of these point estimates to failure of conditional independence, we estimate the breakdown point $c^{*}$ for the conclusion $\text{ATE}\geq 0$, where we use log-wages as our outcome variable. We measure relaxations of conditional independence by our conditional $c$-dependence distance. The estimated breakdown point is $\widehat{c}^{*}=0.041$. Based on this point estimate, for all $x\in\{0,1\}$ and $w\in\operatorname*{supp}(W)$ we can allow the conditional propensity scores $\mathbb{P}(X=x\mid Y_{x}=y,W=w)$ to vary $\pm 4$ percentage points around the observed propensity scores $\mathbb{P}(X=x\mid W=w)$ without changing our conclusion.

Is this a big or small amount of variation? Well, as a baseline, the upper bound on $c$ is about 0.73. This is an estimate of

[TABLE]

Any $c\geq 0.73$ would lead to the no assumptions identified set for ATE. In this sense, $0.041$ is quite small, which would suggest that our results are quite fragile. Next we examine variation in the observed propensity scores as we suggested in Masten and Poirier (2018a). Specifically, we consider the difference between the “full” propensity score and the “leave out variable $k$” propensity score which omits variable $k$: Define

[TABLE]

and

[TABLE]

Using these numbers as a reference, a robust result would have a breakdown point above one or both of the $\overline{c}$’s. In the data, we obtain $\overline{c}_{\texttt{age}}=0.0625$ and $\overline{c}_{\texttt{hhSize}}=0.0403$. The estimated breakdown point $\widehat{c}^{*}=0.041$ is below $\overline{c}_{\texttt{age}}$ and approximately equal to $\overline{c}_{\texttt{hhSize}}$. This latter result suggests that perhaps our conclusion could be considered somewhat robust. Accounting for sampling uncertainty in the breakdown point, however, shows that the true breakdown point may be less than $\overline{c}_{\texttt{hhSize}}$. Overall, this suggests that our conclusion that $\text{ATE}\geq 0$ is not robust to relaxations of full conditional independence.

This argument for judging the plausibility of specific values of $c$ relies on using variation in the observed propensity score to ground our beliefs about reasonable variation in the unobserved propensity scores. The general question here is how one should quantitatively distinguish ‘large’ and ‘small’ relaxations of an assumption. This is an old and ongoing question in the sensitivity analysis literature, and much work remains to be done. For discussions on this point for different measures of deviations or relaxations from independence in various settings, see Rotnitzky et al. (1998), Robins (1999), Imbens (2003), Altonji et al. (2005, 2008), and Oster (2016).

Next consider the parameter $\mathbb{P}(Y_{1}>Y_{0})$. Since we define treatment as not being abducted, this parameter measures the proportion of people who earn higher wages when they are not abducted, compared to when they are abducted. For this parameter, we must make both the full conditional independence assumption and the conditional rank invariance assumption to obtain point identification. Under these assumptions, our point estimate is $0.67$ with a one-sided lower 95% CI of $[0.48,1]$.

Is this point estimate robust to failures of full conditional independence and conditional rank invariance? We examine this question by estimating breakdown frontiers and corresponding confidence bands for the conclusion that $\mathbb{P}(Y_{1}>Y_{0})\geq\underline{p}$. We do this for $\underline{p}=0.1$, $0.25$, $0.5$ as in our Monte Carlo simulations in supplemental appendix C. We do not consider $\underline{p}=0.75$ or $0.9$ since these values are larger than our point estimate under the baseline assumptions; they yield empty estimated robust regions. Besides picking a grid of $\underline{p}$’s a priori, one could let $\underline{p}=\widehat{p}_{0,0}/2$, half the value of the parameter estimated under the baseline point identifying assumptions. In our application this is $0.34$; we omit this choice for brevity. Imbens (2003) suggests a similar choice of cutoff in his approach. We use the same eight ratios of $\varepsilon_{N}/\varepsilon_{N}^{\text{naive}}$ as in our Monte Carlo simulations in supplemental appendix C.

Figure 3 shows the results. As in our earlier plots, the horizontal axis plots $c$, relaxations of full conditional independence, while the vertical axis plots $t$, relaxations of conditional rank invariance. As mentioned earlier, the natural upper bound for $c$ is about $0.73$. Since all of the breakdown frontiers intersect the horizontal axis at much smaller values, we have cut off the part of the overall assumption space with $c\geq 0.2$. Remember that, for the following analysis, it’s valid to examine various $(c,t)$ combinations since we use uniform confidence bands.

First consider the left plot, $\underline{p}=0.1$. Since this is the weakest conclusion of the five we consider, the estimated breakdown frontier and the corresponding robust region are the largest among the three plots. If we impose full conditional independence, then our estimated frontier suggests that we can completely relax conditional rank invariance and still conclude that at least 10% of people benefit from not being forced into military service. Even accounting for sampling uncertainty, we can still draw this conclusion. Moreover, looking at all choices of $\varepsilon_{N}$—not just our selected one—the lowest the vertical intercept ever gets is about 61%. Next suppose we relax full conditional independence. Recall that the maximal relaxation between the observed propensity score and the “leave out variable $k$” propensity scores gave $\overline{c}_{\texttt{age}}=0.0625$ and $\overline{c}_{\texttt{hhSize}}=0.0403$. Both of these numbers are substantially smaller than the horizontal intercept of our selected confidence band. Hence, if we impose full conditional rank invariance, our conclusion that $\mathbb{P}(Y_{1}>Y_{0})\geq 0.1$ is robust to relaxations of full conditional independence. Suppose instead that we think selection on unobservables is at most the largest $\overline{c}$ value, about $0.06$. Then for $c$’s in the range $[0,0.06]$, and accounting for sampling uncertainty, we can still conclude $\mathbb{P}(Y_{1}>Y_{0})\geq 0.1$ so long as at least 30% of the population satisfies rank invariance. Thus we can relax full independence within this range without paying too high a cost in terms of requiring stronger rank invariance assumptions.

If we are willing to restrict selection on unobservables to be smaller than the largest $\overline{c}$ value, then we can allow for larger relaxations of conditional rank invariance. To quantify this trade-off, we can compute the difference between the values of the estimated breakdown frontier at two different points. As a starting point we recommend computing

[TABLE]

where $K$ denotes the number of observed regressors, $\overline{c}_{(K)}$ denotes the largest value of $\overline{c}$, and $\overline{c}_{(K-1)}$ denotes the second largest value. One could also divide this difference by $\overline{c}_{(K)}-\overline{c}_{(K-1)}$ to get a secant line. In our empirical application, $\overline{c}_{(K)}=\overline{c}_{\texttt{age}}$ and $\overline{c}_{(K-1)}=\overline{c}_{\texttt{hhSize}}$. Thus

[TABLE]

The corresponding secant line has slope about $-3$. Thus if we assume selection on unobservables is at most as large as the second largest amount of variation in “leave out variable $k$” propensity scores, we can allow for an additional 6% of the population to violate conditional rank invariance. Put differently, around these values of $c$, allowing latent conditional propensity scores to vary an extra 1 percentage point requires us to impose that an additional 3% of the population must satisfy conditional rank invariance. This rate of substitution generally increases as $c$ gets larger. Our ability to quantify this kind of trade-off between assumptions is a primary goal of our breakdown frontier analysis.

Overall, our results from this top left plot suggest that the conclusion that at least 10% of people benefit from not being forced into military service is robust to relaxations of full conditional independence up to twice the size we see between the observed and leave out variable $k$ propensity scores, depending on how much conditional rank invariance failure we allow. For relaxations of full conditional independence up to the largest value of $\overline{c}$, we can allow up to 70% of the population to deviate from conditional rank invariance, accounting for sampling uncertainty.

Next consider the middle plot, $\underline{p}=0.25$. Since this is a stronger conclusion than the previous one, all the frontiers are shifted towards the origin. Consequently, by construction, this conclusion is not as robust as the other one. Our qualitative conclusions, however, as similar to those obtained for $\underline{p}=0.1$. If we impose full conditional independence we can allow conditional rank invariance to fail for about 70% of the population. Conversely, if we impose full conditional rank invariance, we can allow the latent conditional propensity scores to vary by about 10 percentage points—well beyond the largest observed variation $\overline{c}$. For $\underline{p}=0.25$, we have

[TABLE]

Hence the slope around our observed maximal $\overline{c}$’s is much larger for $\underline{p}=0.25$ as compared to $\underline{p}=0.1$. An important caveat to our conclusions for both $\underline{p}=0.1$ and $\underline{p}=0.25$ is that there is substantial variation in confidence bands as $\varepsilon_{N}$ changes. This point underscores the need for future work on the choice of $\varepsilon_{N}$.

Next consider the right plot, $\underline{p}=0.5$. Here we consider the conclusion that at least half of people benefit from not being forced into military service. If we impose full conditional independence, and accounting for sampling uncertainty, then we can allow conditional rank invariance to fail for about 25% of the population. This is quite large, but it relies on full conditional independence holding exactly. If we also relax conditional independence to $c=0.03$ then we need conditional rank invariance to hold for everyone if we still want to conclude that at least 50% of people benefit from not being forced into military service. $0.03$ is smaller than both $\overline{c}_{\texttt{age}}$ and $\overline{c}_{\texttt{hhSize}}$. Hence we might not be comfortable with such small values of $c$. This suggests the data do not definitively support the conclusion $\mathbb{P}(Y_{1}>Y_{0})\geq 0.5$, even though our point estimate under the baseline assumptions is $0.67$.

In this section we used our breakdown frontier methods to study the robustness of conclusions about ATE and $\mathbb{P}(Y_{1}>Y_{0})$ to failures of conditional independence and conditional rank invariance. We first considered the conclusion that the average treatment effect of not being abducted on log wages is nonnegative. Our point estimates suggest that this conclusion is robust to deviations in unobserved latent propensity scores up to the same value as $\overline{c}_{\texttt{age}}$, which is also about two-thirds as large as $\overline{c}_{\texttt{hhSize}}$; this robustness does not hold up when accounting for sampling uncertainty, however. We then considered the conclusion that at least $\underline{p}$% of people earn higher wages when they are not abducted. This conclusion is robust to large simultaneous relaxations of conditional rank invariance and conditional independence for $\underline{p}=$ 10%. For $\underline{p}=$ 25%, This conclusion continues to be robust to reasonable relaxations, although after accounting for the variation in confidence bands over $\varepsilon_{N}$, this conclusion appears to be more sensitive to conditional independence than to conditional rank invariance. This robustness to rank invariance matches the findings of Heckman et al. (1997), who imposed full independence and studied deviations from rank invariance. In their table 5B they found that, in their empirical application, one could generally conclude that $\mathbb{P}(Y_{1}>Y_{0})$ was at least 50%, regardless of the assumption on rank invariance. In our empirical application our results are not quite as robust to rank invariance failures, which could be because we use a different measure of relaxation of rank invariance, and also because of differences in the empirical applications.

5 Conclusion

Summary

In this paper we advocated the breakdown frontier approach to sensitivity analysis. Given a set of baseline assumptions, this approach defines the population breakdown frontier as the weakest set of assumptions such that a specific conclusion of interest holds. Sample analog estimates and lower uniform confidence bands allow researchers to do inference on this frontier. The area under the confidence band is a quantitative, finite sample measure of the robustness of a conclusion to relaxations of point-identifying assumptions. To examine this robustness, empirical researchers can present these estimated breakdown frontiers and their accompanying confidence bands along with traditional point estimates and confidence intervals obtained under point identifying assumptions. We illustrated this general approach in the context of a treatment effects model, where the robustness of conclusions about ATE and $\mathbb{P}(Y_{1}>Y_{0})$ to relaxations of random assignment and rank invariance are examined. We applied these results in an empirical study of the effect of child soldiering on wages. We found that weak conclusions about $\mathbb{P}(Y_{1}>Y_{0})$ are fairly robust to failures of both rank invariance and random assignment, but stronger conclusions are more sensitive to relaxations of random assignment.

Breakdown frontier analysis for other models and other relaxations

As discussed in section 1, breakdown frontier analysis can in principle be done in most models. In that section we outlined the six main steps required for any breakdown frontier analysis. In this paper we illustrated this general approach by studying a single important and widely used model: the potential outcomes model with a binary treatment. In future work it would be helpful to perform breakdown frontier analyses in other models. In particular, it may be possible to do breakdown frontier analyses in a large class of models by using the general identification analysis in Chesher and Rosen (2017) or Torgovitsky (2018).

A key conceptual step in any breakdown frontier analysis is deciding how to define the indexed classes of assumptions such that the magnitude of the relaxation can be reasonably interpreted. This is not easy, and will generally depend on the model, the specific kind of assumption being relaxed, and the empirical context. Moreover, this choice may affect our findings: A conclusion can be robust with respect to one measure of relaxation but not another. Thus one goal of future research is to explore this space of assumption relaxations, to understand their substantive interpretations, and to chart their implications for the robustness of empirical findings. In Masten and Poirier (2016) we have already compared three different measures of relaxation of the random assignment assumption, including the one used here. We further studied quantile independence, a common relaxation of random assignment, in Masten and Poirier (2018b). In the present paper, we also used a general method for spanning two discrete assumptions by defining a (1$-t$)-percent relaxation, as we did with rank invariance. But much work still remains to be done.

Appendix A Related literature

In this section, we review the related literature. We begin with the identification literature on breakdown points; as mentioned earlier, here we use “breakdown” in the same sense as Horowitz and Manski’s (1995) identification breakdown point. This breakdown point idea goes back to the one of the earliest sensitivity analyses, performed by Cornfield et al. (1959). They essentially asked how much correlation between a binary treatment and an unobserved binary confounder must be present to fully explain an observed correlation between treatment and a binary outcome, in the absence of any causal effects of treatment. This level of correlation between treatment and the confounder is a kind of breakdown point for the conclusion that some causal effects of treatment are nonzero. Their approach was substantially generalized by Rosenbaum and Rubin (1983), which is discussed in detail in chapter 22 of Imbens and Rubin (2015). Neither Cornfield et al. (1959) nor Rosenbaum and Rubin (1983) formally defined breakdown points.

Horowitz and Manski (1995) gave the first formal definition and analysis of breakdown points. They studied a “contaminated sampling” model, where one observes a mixture of draws from the distribution of interest and draws from some other distribution. An upper bound $\lambda$ on the unknown mixing probability indexes identified sets for functionals of the distribution of interest. They focus on a single conclusion: That this functional is not equal to its logical bounds. They then define the breakdown point $\lambda^{*}$ as the largest $\lambda$ such that this conclusion holds. Put differently, $\lambda^{*}$ is the largest mixing probability we can allow while still obtaining a nontrivial identified set for our parameter of interest. They also relate this “identification breakdown point” to the earlier breakdown point concepts studied in the robust statistics literature (e.g., Hampel et al. 1986 pages 96–98 and Huber and Ronchetti 2009 section 1.4 and chapter 11).

More generally, much work by Manski distinguishes between informative and noninformative bounds (which the literature also sometimes calls tight and non-tight bounds; see section 7.2 of Ho and Rosen 2017). The breakdown point is the boundary between the informative and noninformative cases. For example, see his analysis of bounds on quantiles with missing outcome data on page 40 of Manski (2007). There the identification breakdown point for the $\tau$th quantile occurs when $\max\{\tau,1-\tau\}$ is the proportion of missing data. Similar discussions are given throughout the book.

Stoye (2005, 2010) generalizes the formal identification breakdown point concept by noting that breakdown points can be defined for any claim about the parameter of interest. He then studies a specific class of relaxations of the missing-at-random assumption in a model of missing data. Kline and Santos (2013) study a different class of relaxations of the missing-at-random assumption and also define a breakdown point based on that class.

While all of these papers study a scalar breakdown point, Imbens (2003) studies a model of treatment effects where deviations from conditional random assignment are parameterized by two numbers $r=(r_{1},r_{2})$. His parameter of interest $\theta(r)$ is point identified given a fixed value of $r$. Imbens’ figures 1–4 essentially plot estimated level sets of this function $\theta(r)$, in a transformed domain. While suggestive, these level sets do not generally have a breakdown frontier interpretation. This follows since non-monotonicities in the function $\theta(r)$ lead to level sets which do not always partition the space of sensitivity parameters into two connected sets in the same way that our breakdown frontier does.

Manski and Pepper (2018) also study a model where relaxations of baseline assumptions are parameterized by a vector of numbers $r$. Unlike Imbens, however, they derive identified sets indexed by $r$. These sets are weakly increasing (in the set inclusion order) in each component of $r$, and hence the non-monotonicity issue does not arise. For a two-dimensional relaxation, their table 2 presents identified sets as a function of a grid of $r=(r_{1},r_{2})$ values. The boundary between the italicized identified sets in that table and the non-highlighted sets is essentially a discrete approximation to the breakdown frontier in their model, for the claim that the parameter of interest is positive. Similarly, the boundary between the bold identified sets in that table and the non-highlighted sets is essentially a discrete approximation to the breakdown frontier in their model, for the claim that the parameter of interest is negative.

Neither Horowitz and Manski (1995) nor Stoye (2005, 2010) discuss estimation or inference of breakdown points. Imbens (2003) estimates his level sets in an empirical application, but does not discuss inference. Manski and Pepper (2018) also do not discuss estimation of or inference on breakdown frontiers, although inference in their setting is conceptually complicated—see their discussion on pages 234–235. Kline and Santos (2013), on the other hand, is the first and only paper we’re aware of that explicitly suggests doing inference on a breakdown point. We build on their work by proposing to do inference on the multi-dimensional breakdown frontier. This allows us to study the trade-off between different assumptions in drawing conclusions. They do study something they call a ‘breakdown curve’, but this is a collection of scalar breakdown points for many different claims of interest, analogous to the collection of frontiers presented in figures 2a, 2b, and 2c. Inference on a frontier rather than a point also raises additional issues they did not discuss; see our supplemental appendix A for more details. Moreover, we study a model of treatment effects while they look at a model of missing data, hence our identification analysis is different.

Building on Horowitz and Manski (1995), Kreider et al. (2012) combine a continuous relaxation sensitivity analysis for assumptions regarding measurement error with various discrete relaxations of assumptions regarding treatment selection. This allows them to study the interaction between these two kinds of assumptions in drawing conclusions. For inference they present confidence intervals for partially identified parameters for a variety of values of the relaxations, rather than doing inference on breakdown frontiers. See Gundersen et al. (2012) and Kreider et al. (2016) for further examples of identification analysis combining discrete and continuous relaxations.

Our breakdown frontier is a known functional of the distribution of outcomes given treatment and covariates and the observed propensity scores. This functional is not Hadamard differentiable, however, which prevents us from applying the standard functional delta method to obtain its asymptotic distribution. Instead, we show that it is Hadamard directionally differentiable, which allows us to apply the results of Fang and Santos (2019). We then use the numerical bootstrap of Dümbgen (1993) and Hong and Li (2018) to construct our confidence bands. For other applications of Hadamard directional differentiability, see Lee and Bhattacharya (2016), Kaido (2016), and Hansen (2017).

Our identification analysis builds on two strands of literature. First is the literature on relaxing statistical independence assumptions. There is a large literature on this, including important work by Rosenbaum and Rubin (1983), Robins et al. (2000), and Rosenbaum (1995, 2002). We apply results from our paper Masten and Poirier (2018a), which discusses that literature in more detail. In that paper we did not study estimation or inference. Second is the literature on identification of the distribution of treatment effects $Y_{1}-Y_{0}$, especially without the rank invariance assumption. In their introduction, Fan et al. (2017) provide a comprehensive discussion of this literature; also see Abbring and Heckman (2007) section 2. Here we focus on the papers most related to our sensitivity analysis. Heckman et al. (1997) performed a sensitivity analysis to the rank invariance assumption by fixing the value of Kendall’s $\tau$ for the joint distribution of potential outcomes, and then varying $\tau$ from $-1$ to $1$; see tables 5A and 5B. Their analysis is motivated by a search for breakdown points, as evident in their section 4 title, “How far can we depart from perfect dependence and still produce plausible estimates of program impacts?” Nonetheless, they do not formally define identified sets for parameters given their assumptions on Kendall’s $\tau$, and they do not formally define a breakdown point. Moreover, they do not suggest estimating or doing inference on breakdown points. Gechter (2016) performs a sensitivity analysis to the rank invariance assumption by fixing a lower bound on the value of Spearman’s $\rho$. Under this assumption he derives the identified set for a certain average treatment effect. He then studies estimation of and inference on this set for a fixed value of the sensitivity parameter. Fan and Park (2009) provide formal identification results for the joint cdf of potential outcomes and the distribution of treatment effects under the known Kendall’s $\tau$ assumption. They also discuss how to extend those results to known Spearman’s $\rho$ in their remark 1. They provide estimation and inference methods for their bounds, but do not study breakdown points. Finally, none of these papers studies the specific relaxation of rank invariance we consider (as defined in section 2).

In this section we have focused narrowly on the papers most closely related to ours. We situate our work more broadly in the literature on inference in sensitivity analyses in supplemental appendix A. In that section we also briefly discuss Bayesian inference, although we use frequentist inference in this paper.

Appendix B Estimation and inference with continuous covariates

The estimation and inference theory in section 3 assumes that the covariates $W$ are discretely distributed (via A4). Those results are nonparametric in the sense that they do not impose any restrictions on the conditional distribution of $Y\mid X,W$ or on the propensity score $p_{x|w}$. But they rule out continuous covariates. In this section, we briefly discuss how to do estimation and inference with continuous covariates.

When some components of $W$ are continuously distributed, a simple solution is to discretize $W$ and then apply the previous estimator. Alternatively, one can smooth over different covariate values. This can be done using parametric, semiparametric, or nonparametric estimators.

For example, especially if the dimension of $W$ is large, one could use the usual logit propensity score estimator

[TABLE]

where $\Lambda(a)=\exp(a)/(1+\exp(a))$ is the standard logit cdf and $\widehat{\beta}$ are the maximum likelihood estimated index coefficients. The conditional quantile function $Q_{Y\mid X,W}(\tau\mid x,w)$ can be estimated by a linear quantile regression of $Y$ on $(1,X,W)$, so that

[TABLE]

where $\widehat{\gamma}(\tau)$ are estimated linear quantile regression coefficients.

Using these parametric estimators, define

[TABLE]

and

[TABLE]

as before. Since the asymptotic properties of $\widehat{\beta}$ and $\widehat{\gamma}(\cdot)$ are well known, it should be feasible to derive the asymptotic distribution of the functionals we consider in section 3. Alternatively, one could use semiparametric or nonparametric estimators of the propensity score $p_{x|w}$ and the conditional quantile function $Q_{Y\mid X,W}$. Again, such first step estimators can be plug-ins to obtain estimates of the various bounds we consider in section 3. We leave a full analysis of the asymptotic properties of these estimators to future work.

Appendix C Proofs

Proofs for section 2

Proof of theorem 1.

Let $F_{1}(\cdot\mid w)$ and $F_{0}(\cdot\mid w)$ be any strictly increasing cdfs conditional on $W=w$ for any $w\in\operatorname*{supp}(W)$. Suppose $(Y_{1},Y_{0})\mid W$ have joint cdf

[TABLE]

Then

[TABLE]

For fixed distributions $(F_{1}(\cdot\mid w),F_{0}(\cdot\mid w))$, the first integral is the probability that $\{Y_{1}-Y_{0}\leq z\}$ given $W=w$, where $(Y_{1},Y_{0})\mid W$ are random variables that satisfy conditional rank invariance. Hence for these random variables the corresponding conditional ranks are equal almost surely: Conditional on $W=w$, $U_{1}=U_{0}$ a.s., let $U\sim\text{Unif}[0,1]$ denote this almost sure common random variable. Using A1.1, we can thus write

[TABLE]

and therefore

[TABLE]

Makarov (1981) derived sharp bounds on

[TABLE]

Applying these bounds yields

[TABLE]

Therefore, for given $w\in\operatorname*{supp}(W)$ and given $(F_{1}(\cdot\mid w),F_{0}(\cdot\mid w))$, sharp bounds for $\mathbb{P}(Y_{1}-Y_{0}\leq z\mid W=w)$ are given by

[TABLE]

where

[TABLE]

and

[TABLE]

Define the first order stochastic dominance ordering as follows: For two cdfs $F$ and $G$, let $F\preceq_{\textsc{fsd}}G$ if $F(t)\geq G(t)$ for all $t\in\mathbb{R}$. All of the following statements refer to this ordering. For any fixed $F_{1}(\cdot\mid w)$,

[TABLE]

That is, the lower bound function $\underline{\theta}(F_{1}(\cdot\mid w),F_{0}(\cdot\mid w))$ is weakly increasing in $F_{0}(\cdot\mid w)$. This can be shown in two steps. First, the expression

[TABLE]

is weakly increasing in $F_{0}(\cdot\mid w)$ since, for $\tilde{F}_{0}(\cdot\mid w)\preceq_{\textsc{fsd}}F_{0}(\cdot\mid w)$, we have $\tilde{F}_{0}^{-1}(u\mid w)\leq F_{0}^{-1}(u\mid w)$ for $u\in(0,1)$, and therefore,

[TABLE]

Second, the expression

[TABLE]

is weakly increasing in $F_{0}(\cdot\mid w)$ since the supremum and maximum operators are weakly increasing. Thus both components of $\underline{\theta}$ are weakly increasing in $F_{0}(\cdot\mid w)$. Therefore their linear combination is also weakly increasing in $F_{0}(\cdot\mid w)$.

We can similarly show that $\underline{\theta}(F_{1}(\cdot\mid w),F_{0}(\cdot\mid w))$ is weakly decreasing in $F_{1}(\cdot\mid w)$. Thus substituting

[TABLE]

yields the lower bound $\underline{\text{CDTE}}(z,c,t\mid w)$. The upper bound function $\overline{\theta}(F_{1}(\cdot\mid w),F_{0}(\cdot\mid w))$ is also weakly increasing in $F_{0}(\cdot\mid w)$ and weakly decreasing in $F_{1}(\cdot\mid w)$. Thus substituting

[TABLE]

yields the upper bound $\overline{\text{CDTE}}(z,c,t\mid w)$. In making these substitutions we applied proposition 2 from Masten and Poirier (2018a). In that paper we defined functions $F_{Y_{x}\mid W}^{c}(\cdot\mid w;\epsilon,\eta)$, which we now use to sharpness of the DTE bounds.

Substitute

[TABLE]

into the bound functionals and continuously vary $\epsilon$ between $[0,1]$. Note that we let $\eta=0$ since $c<\min\{p_{1|w},p_{0|w}\}$. By continuity of $\underline{\theta}(\cdot,\cdot)$ and $\overline{\theta}(\cdot,\cdot)$ in their arguments and continuity of $(F_{Y_{1}\mid W}^{c}(\cdot\mid w;\epsilon,0),F_{Y_{0}\mid W}^{c}(\cdot\mid w;1-\epsilon,0))$ in $\epsilon$, the intermediate value theorem implies that every element between the bounds can be attained.

By integrating these CDTE bounds over the marginal distribution of $W$, we obtain the DTE bounds:

[TABLE]

Sharpness of these bounds results from the sharpness of the CDTE bounds for every $w\in\operatorname*{supp}(W)$ and the joint attainability of

[TABLE]

and of

[TABLE]

∎

Proofs for section 3

The following lemma shows that $(\widehat{F}_{Y\mid X,W}(\cdot\mid\cdot,\cdot),\widehat{p}_{(\cdot|\cdot)},\widehat{q}_{(\cdot)})$ converges uniformly in $y$, $x$, and $w$ to a mean-zero Gaussian process. This result follows by applying the delta method.

Lemma 5**.**

Suppose A3 and A4 hold. Then

[TABLE]

a mean-zero Gaussian process in $\ell^{\infty}(\mathbb{R}\times\{0,1\}\times\operatorname*{supp}(W),\mathbb{R}^{3})$ with continuous paths and covariance kernel equal to

[TABLE]

Proof of lemma 5.

By a second order Taylor expansion,

[TABLE]

By standard bracketing entropy results (e.g., example 19.6 on page 271 of van der Vaart 2000) the function classes $\{\mathbbm{1}(Y\leq y)\mathbbm{1}(X=x)\mathbbm{1}(W=w):y\in\mathbb{R},x\in\{0,1\},w\in\operatorname*{supp}(W)\}$ and $\{\mathbbm{1}(X=x)\mathbbm{1}(W=w):x\in\{0,1\},w\in\operatorname*{supp}(W)\}$ are both $P$-Donsker. Hence the residual is of order $O_{p}(N^{-1})$ uniformly in $(y,x,w)\in\mathbb{R}\times\{0,1\}\times\operatorname*{supp}(W)$. Combining this with Slutsky’s theorem we get the uniform over $y$, $x$, and $w$ asymptotically linear representation

[TABLE]

By the same bracketing entropy arguments, the class

[TABLE]

is $P$-Donsker and hence $\sqrt{N}(\widehat{F}_{Y\mid X,W}(\cdot\mid\cdot,\cdot)-F_{Y\mid X,W}(\cdot\mid\cdot,\cdot))$ converges in distribution to a mean-zero Gaussian process with continuous paths.

A similar argument yields the asymptotically linear representations

[TABLE]

and

[TABLE]

The covariance kernel $\mathbf{\Sigma}_{1}$ can be calculated as follows:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

∎

Lemma 6** (Chain Rule for Hadamard directionally differentiable functions).**

Let $\mathbb{D}$, $\mathbb{E}$, and $\mathbb{F}$ be Banach spaces with norms $\|\cdot\|_{\mathbb{D}}$, $\|\cdot\|_{\mathbb{E}}$, and $\|\cdot\|_{\mathbb{F}}$. Let $\mathbb{D}_{\phi}\subseteq\mathbb{D}$ and $\mathbb{E}_{\psi}\subseteq\mathbb{E}$. Let $\phi:\mathbb{D}_{\phi}\to\mathbb{E}_{\psi}$ and $\psi:\mathbb{E}_{\psi}\to\mathbb{F}$ be functions. Let $\theta\in\mathbb{D}_{\phi}$ and $\phi$ be Hadamard directionally differentiable at $\theta$ tangentially to $\mathbb{D}_{0}\subseteq\mathbb{D}$. Let $\psi$ be Hadamard directionally differentiable at $\phi(\theta)$ tangentially to the range $\phi_{\theta}^{\prime}(\mathbb{D}_{0})\subseteq\mathbb{E}_{\psi}$. Then, $\psi\circ\phi:\mathbb{D}_{\phi}\to\mathbb{F}$ is Hadamard directionally differentiable at $\theta$ tangentially to $\mathbb{D}_{0}$ with Hadamard directional derivative equal to $\psi_{\phi(\theta)}^{\prime}\circ\phi_{\theta}^{\prime}$.

This result is a version of proposition 3.6 in Shapiro (1990), who omits the proof. We give the proof here because this result is key to our paper.

Proof of lemma 6.

Let $\{h_{n}\}_{n\geq 1}$ be in $\mathbb{D}$ and $h_{n}\to h\in\mathbb{D}_{0}$. By Hadamard directional differentiability of $\phi$ tangentially to $\mathbb{D}_{0}$

[TABLE]

as $n\to\infty$ for any $t_{n}\searrow 0$. That is,

[TABLE]

where $\phi_{\theta}^{\prime}\in\phi_{\theta}^{\prime}(\mathbb{D}_{0})$. Therefore, by Hadamard directional differentiability of $\psi$, we have

[TABLE]

By Hadamard directional differentiability of $\phi$ at $\theta$ and $\psi$ at $\phi(\theta)$, $\phi^{\prime}_{\theta}$ and $\psi^{\prime}_{\phi(\theta)}$ are continuous mappings. Hence their composition $\psi^{\prime}_{\phi(\theta)}\circ\phi_{\theta}^{\prime}$ is continuous. This combined with our derivations above imply that $\psi\circ\phi$ is Hadamard directionally differentiable tangentially to $\mathbb{D}_{0}$ at $\theta$. ∎

Proof of lemma 1.

Let $\theta_{0}=(F_{Y\mid X,W}(\cdot\mid\cdot,\cdot),p_{(\cdot|\cdot)},q_{(\cdot)})$ and $\widehat{\theta}=(\widehat{F}_{Y\mid X,W}(\cdot\mid\cdot,\cdot),\widehat{p}_{(\cdot|\cdot)},\widehat{q}_{(\cdot)})$. For fixed $y$ and $c$ define the mapping

[TABLE]

by

[TABLE]

where $\theta^{(j)}$ is the $j$th component of $\theta$. Note that

[TABLE]

The maps $(a_{1},a_{2})\mapsto\min\{a_{1},a_{2}\}$ and $(a_{1},a_{2})\mapsto\max\{a_{1},a_{2}\}$ are Hadamard directionally differentiable with Hadamard directional derivatives at $(a_{1},a_{2})$ equal to

[TABLE]

and

[TABLE]

respectively, where $h\in\mathbb{R}^{2}$; for example, see equation (18) in Fang and Santos (2015). The mapping $\phi_{1}$ is comprised of compositions of these $\min$ and $\max$ operators, along with four other functions. We can show that these four mappings are ordinary Hadamard differentiable. Here we compute these Hadamard derivatives with respect to $\theta$:

[TABLE]

All these derivatives are well defined at $\theta_{0}$ because $\theta^{(2)}_{0}(x,w)=p_{x|w}>\overline{C}\geq c$. With this notation, we can write the functional $\phi_{1}$ as

[TABLE]

By the chain rule (lemma 6), the map $\phi_{1}$ is Hadamard directionally differentiable at $\theta_{0}$ with Hadamard directional derivative evaluated at $\theta_{0}$ equal to

[TABLE]

By lemma 5, $\sqrt{N}(\widehat{\theta}(y,x,w)-\theta_{0}(y,x,w))\rightsquigarrow\mathbf{Z}_{1}(y,x,w)$. Hence we can use the delta method for Hadamard directionally differentiable functions (see theorem 2.1 in Fang and Santos 2019) to find that

[TABLE]

This result holds uniformly over any finite grid of values for $y\in\mathbb{R}$ and $c\in\mathcal{C}$ by considering the Hadamard directional differentiability of a vector of these mappings indexed at different values of $y$ and $c$, which yields the process $\mathbf{Z}_{2}(y,x,w,c)$. ∎

Proof of lemma 2.

Let $\mathcal{S}=\{(y,x,w)\in\mathbb{R}^{2}:y\in[\underline{y}_{x}(w),\overline{y}_{x}(w)],x\in\{0,1\},w\in\operatorname*{supp}(W)\}$. Let $\mathscr{D}(\mathcal{S})\subset\ell^{\infty}(\mathcal{S})$ denote the set of functions that are càdlàg in the first argument for each $x\in\{0,1\}$ and $w\in\operatorname*{supp}(W)$. Define the mapping

[TABLE]

by

[TABLE]

By A1, A3, A5, and lemma 21.4(ii) in van der Vaart (2000) this mapping is Hadamard differentiable at $\theta_{0}$ tangentially to $\mathscr{C}(\mathcal{S})\times\ell^{\infty}(\{0,1\}\times\operatorname*{supp}(W))\times\ell^{\infty}(\operatorname*{supp}(W))$, where $\mathscr{C}(\mathcal{S})\subset\ell^{\infty}(\mathcal{S})$ is the set functions that are continuous in the first argument for each $x\in\{0,1\}$ and $w\in\operatorname*{supp}(W)$. Its Hadamard derivative at $\theta_{0}=(F_{Y\mid X,W}(\cdot\mid\cdot,\cdot),p_{(\cdot|\cdot)},q_{(\cdot)})$ is

[TABLE]

By the functional delta method and theorem 7.3.3 part (iii) of Bickel and Doksum (2015),

[TABLE]

where $\tilde{\mathbf{Z}}_{3}$ is a mean-zero Gaussian process in $\ell^{\infty}((0,1)\times\{0,1\}\times\operatorname*{supp}(W),\mathbb{R}^{2})$ with uniformly continuous paths.

Now define the mapping

[TABLE]

by

[TABLE]

Then

[TABLE]

We will show that $\phi_{2}$ is Hadamard differentiable tangentially to the space $\mathscr{C}_{U}((0,1)\times\{0,1\}\times\operatorname*{supp}(W))\times\ell^{\infty}(\{0,1\}\times\operatorname*{supp}(W))$, where $\mathscr{C}_{U}(\mathcal{A})$ denotes the set of uniformly continuous functions on $\mathcal{A}$. The Hadamard derivative of the first component of $\phi_{2}$ evaluated at $\psi_{0}\equiv\tilde{\phi}_{2}(\theta_{0})$ is

[TABLE]

To see this, a Taylor expansion gives

[TABLE]

using the fact that $\psi_{0}^{(1)}(\tau,x,w)=Q_{Y\mid X}(\tau\mid x,w)$ is continuously differentiable in $\tau$ by assumption A5.2, and noting that term $a_{n}(x,w)$ satisfies $|a_{n}(x,w)|\leq|t_{n}h_{n}^{(2)}(x,w)|=O(t_{n})$. Next,

[TABLE]

where all three suprema are taken over $\tau\in(0,1),x\in\{0,1\},w\in\operatorname*{supp}(W),c\in[0,\overline{C}]$. The last inequality follows from uniform continuity of $h^{(1)}$. The last line follows from uniform convergence of $h_{n}$ to $h$.

Similarly, we have that

[TABLE]

by uniform continuity of $\psi_{0}^{(1)\prime}$ (implied by A5.2) and by $a_{n}(x,w)=o(1)$. Again, the sup is over $\tau\in(0,1),x\in\{0,1\},w\in\operatorname*{supp}(W),c\in[0,\overline{C}]$. Therefore $\phi_{2}^{(1)}$ is Hadamard differentiable tangentially to the space of uniformly continuous functions. A similar argument can be made for $\phi_{2}^{(2)}$. By composition, $\phi_{2}\circ\tilde{\phi}_{2}$ is Hadamard differentiable tangentially to $\mathscr{C}(\mathcal{S})$.

By the functional delta method and the fact that $\tilde{\mathbf{Z}}_{3}(y,x,w)$ has uniformly continuous paths, we have that

[TABLE]

a mean-zero Gaussian process with continuous paths in $\tau\in(0,1)$ and $c\in[0,\overline{C}]$. ∎

Proof of proposition 1.

Consider the lower CQTE bound of equation (4) as a function of $c$. Its first component is the lower bound of the conditional quantile of $Y_{1}\mid W=w$. By assumption A5.2, the derivative of that conditional quantile with respect to $c$ equals

[TABLE]

The second component of the lower CQTE bound is the upper bound of the conditional quantile of $Y_{0}\mid W=w$. The derivative of that conditional quantile with respect to $c$ equals

[TABLE]

Moreover, these derivatives are bounded away from zero and infinity uniformly over $c\in(0,\overline{C}]$. This implies that the derivative of the CQTE is negative and uniformly bounded away from zero.

Next recall that

[TABLE]

Its derivative with respect to $c$ exists by the dominated convergence theorem (by A1 and A5). Moreover, it is bounded away from zero for all $c\in(0,\overline{C}]$. By taking another expectation over the marginal distribution of $W$, $\partial\underline{\text{ATE}}(c)/\partial c$ exists (by A4), is negative, and is bounded away from zero for all $c\in(0,\overline{C}]$.

$c^{*}$ is defined implicitly by $\underline{\text{ATE}}(c^{*})=\mu$. We have shown that the function $\underline{\text{ATE}}(c)$ satisfies the assumptions of lemma 21.3 on page 306 of van der Vaart (2000). Thus the mapping $\underline{\text{ATE}}(\cdot)\mapsto c^{*}$ is Hadamard differentiable tangentially to the set of càdlàg functions on $(0,\overline{C}]$ with derivative

[TABLE]

By the discussion following lemma 2, $\sqrt{N}(\widehat{\underline{\text{ATE}}}(c)-\underline{\text{ATE}}(c))$ converges in distribution to a random element of $\ell^{\infty}([0,\overline{C}])$ with continuous paths.

Let

[TABLE]

We can then apply the functional delta method to see that $\sqrt{N}(\tilde{c}^{*}-c^{*})$ converges in distribution to a Gaussian variable we denote by $\mathbf{Z}_{4}$.

Since $c^{*}\in(0,\overline{C}]$ and by monotonicity of $\underline{\text{ATE}}(\cdot)$, we have $\underline{\text{ATE}}(\overline{C})\leq\mu$. By $\sqrt{N}$-convergence of the ATE bounds,

[TABLE]

Therefore, the set $\{c\in[0,\overline{C}]:\widehat{\underline{\text{ATE}}}(c)\leq\mu\}$ is non-empty with probability approaching one. This implies that $\tilde{c}^{*}\in[0,\overline{C}]$ with probability approaching one, and therefore $\mathbb{P}(\tilde{c}^{*}=\widehat{c}^{*})$ also approaches one as $N\rightarrow\infty$. Using these results, we obtain

[TABLE]

∎

The following result extends proposition 2(i) of Chernozhukov et al. (2010) to allow for input functions which are directionally differentiable, but not fully differentiable, at one point. It can be extended to allow for multiple points of directional differentiability, but we omit this since we do not need it for our application.

Lemma 7**.**

Let $\theta_{0}(u,c,w)=(\theta_{0}^{(1)}(u,c,w),\theta_{0}^{(2)}(u,c,w))$ where for $j\in\{1,2\}$ we have that $\theta_{0}^{(j)}(u,c,w)$ is strictly increasing in $u\in[0,1]$, bounded above and below, and differentiable everywhere except at $u=u^{*}$, where it is directionally differentiable. Further, assume that the two components satisfy A6. Then, for fixed $z\in\mathbb{R}$, the mapping $\phi_{3}:\ell^{\infty}((0,1)\times\operatorname*{supp}(W)\times\mathcal{C},\mathbb{R}^{2})\rightarrow\ell^{\infty}(\operatorname*{supp}(W)\times\mathcal{C},\mathbb{R}^{2})$ defined by

[TABLE]

is Hadamard directionally differentiable tangentially to $\mathscr{C}((0,1)\times\operatorname*{supp}(W)\times\mathcal{C},\mathbb{R}^{2})$ with Hadamard directional derivative given by equations (17) and (18) below.

Proof of lemma 7.

For clarity we suppress the dependence on $w$ in the expressions below. Uniformity of convergence over $w\in\operatorname*{supp}(W)$ follows from the discreteness of $\operatorname*{supp}(W)$ (assumption A4). Our proof follows that of proposition 2(i) in Chernozhukov et al. (2010). Let

[TABLE]

denote the set of roots to the equation $\theta_{0}^{(1)}(u,c)=z$ for fixed $z$ and $c$. By A6.1 this set contains a finite number of elements. We denote these by

[TABLE]

A6.1 also implies that $\mathcal{U}_{1}(c)\cap\mathcal{U}^{*}_{1}(c)=\emptyset$ for any $c\in\mathcal{C}$.

We will show the first component of the Hadamard directional derivative is given by

[TABLE]

where $h\in\mathscr{C}((0,1)\times\mathcal{C})$.

First suppose $u^{*}\notin\mathcal{U}_{1}(c)$ for any $c\in\mathcal{C}$. In this case we can apply proposition 2(i) of Chernozhukov et al. (2010) directly to obtain

[TABLE]

for any $c\in\mathcal{C}$, where $t_{n}\searrow 0$, $h_{n}\in\ell^{\infty}((0,1)\times\mathcal{C})$, and

[TABLE]

as $n\rightarrow\infty$. Hence

[TABLE]

a linear map in $h$.

Now suppose $u^{*}\in\mathcal{U}_{1}(c)$ for some $c\in\mathcal{C}$. Without loss of generality, let $u_{1}^{(1)}(c)=u^{*}$. Let $B_{\epsilon}(u)$ denote a ball of radius $\epsilon$ centered at $u$. By equation (A.1) in Chernozhukov et al. (2010), for any $\delta>0$ there exists an $\epsilon>0$ and a large enough $n$ such that

[TABLE]

Likewise, for any $\delta>0$ there exists $\epsilon>0$ and large enough $n$ such that

[TABLE]

The $k=1$ element in the first sum is

[TABLE]

$\theta_{0}(u,c)$ is absolutely continuous in $u$ and, by the change of variables formula for absolutely continuous functions, the transformation $z^{\prime}=\theta_{0}(u,c)$ implies that this $k=1$ term is

[TABLE]

where $J_{1}$ is the image of $B_{\epsilon}(u^{*})$ under $\theta_{0}(\cdot,c)$ and the change of variables follows from the monotonicity of $\theta_{0}$ in $B_{\epsilon}(u^{*})$ for small enough $\epsilon$ (this monotonicity follows from A6.1, which implies that the derivative of $\theta_{0}$ changes sign a finite number of times). The closed interval $[z,z-t_{n}(h(u^{*},c)-\delta)]$ should be interpreted as $[z-t_{n}(h(u^{*},c)-\delta),z]$ when $z-t_{n}(h(u^{*},c)-\delta)<z$. Next consider three cases:

When $h(u^{*},c)>0$, the interval $[z,z-t_{n}(h(u^{*},c)-\delta)]$ has the form $[z-\psi_{n},z]$ for an arbitrarily small $\psi_{n}>0$. Therefore, the denominator $|\partial_{u}\theta_{0}(\theta_{0}^{-1}(z^{\prime},c),c)|$ converges to $|\partial^{-}_{u}\theta_{0}(u^{*},c)|$ as $n\rightarrow\infty$, by continuous differentiability on $(0,u^{*})$ and directional differentiability as $u=u^{*}$ and by $\theta_{0}^{-1}(z^{\prime},c)=u^{*}+o(1)$. This holds by $z^{\prime}\in[z-t_{n}(h(u^{*},c)-\delta),z]$, an interval shrinking to $\{z\}$. Therefore,

[TABLE]

By a similar argument,

[TABLE]

Letting $\delta>0$ be arbitrarily small and by the squeeze theorem, we obtain

[TABLE] 2. 2.

When $h(u^{*},c)<0$, the interval $[z,z-t_{n}(h(u^{*},c)-\delta)]$ is of the form $[z,z+\psi_{n}]$ for arbitrarily small $\psi_{n}>0$. Using the same argument as in case 1, $|\partial_{u}\theta_{0}(\theta_{0}^{-1}(z^{\prime},c),c)|$ converges to $|\partial^{+}_{u}\theta_{0}(u^{*},c)|$ as $n\rightarrow\infty$. Therefore, proceeding as in the previous case, we obtain that

[TABLE] 3. 3.

When $h(u^{*},c)=0$, this $k=1$ term converges to zero.

Combining these three cases into a single expression we find that

[TABLE]

This expression coincides with the Hadamard derivative under continuous differentiability at $u=u^{*}$, since that implies $\partial^{-}_{u}\theta_{0}(u^{*},c)=\partial^{+}_{u}\theta_{0}(u^{*},c)$. It follows from the remainder of the proof in Chernozhukov et al. (2010) that

[TABLE]

where $\|\cdot\|_{e}$ is the Euclidean norm, and where $\phi^{(1)\prime}_{3,\theta_{0}}$ is defined in equation (17). Note that $\phi^{(1)\prime}_{3,\theta_{0}}$ is continuous in $h$, and therefore it is a Hadamard directional derivative.

That completes our analysis of the first component of the Hadamard directional derivative of $\phi_{3}$ with respect to $\theta$ at $\theta_{0}$. By similar arguments, the second component is

[TABLE]

∎

Proof of lemma 3.

Let

[TABLE]

Therefore

[TABLE]

By lemma 2,

[TABLE]

a mean-zero Gaussian processes in $\ell^{\infty}((0,1)\times\operatorname*{supp}(W)\times\mathcal{C},\mathbb{R}^{2})$ with continuous paths.

By lemma 7 with $u^{*}=1/2$, the mapping $\phi_{3}$ is Hadamard directionally differentiable tangentially to $\mathscr{C}((0,1)\times\operatorname*{supp}(W)\times\mathcal{C},\mathbb{R}^{2})$. By the functional delta method for Hadamard directionally differentiable functions (e.g., theorem 2.1 in Fang and Santos 2019), we obtain

[TABLE]

a tight random element of $\ell^{\infty}(\operatorname*{supp}(W)\times\mathcal{C},\mathbb{R}^{2})$. ∎

The following lemma shows that the sup operator is Hadamard directionally differentiable. It is a very minor extension of lemma B.1 in Fang and Santos (2015), where we take the supremum over just one of two arguments.

Lemma 8**.**

Let $\mathcal{A}$ and $\mathcal{C}$ be compact subsets of $\mathbb{R}$. Define the map $\phi:\ell^{\infty}(\mathcal{A}\times\mathcal{C})\rightarrow\ell^{\infty}(\mathcal{C})$ by

[TABLE]

Let

[TABLE]

be a set-valued function. Then $\phi$ is Hadamard directionally differentiable tangentially to $\mathscr{C}(\mathcal{A}\times\mathcal{C})$ at any $\theta\in\mathscr{C}(\mathcal{A}\times\mathcal{C})$, and $\phi_{\theta}^{\prime}:\mathscr{C}(\mathcal{A}\times\mathcal{C})\rightarrow\mathscr{C}(\mathcal{C})$ is given by

[TABLE]

for any $h\in\mathscr{C}(\mathcal{A}\times\mathcal{C})$.

Proof of lemma 8.

This proof follows that of Lemma B.1 in Fang and Santos (2015). Let $t_{n}\searrow 0$, and $h_{n}\in\ell^{\infty}(\mathcal{A}\times\mathcal{C})$ such that

[TABLE]

for $h\in\mathscr{C}(\mathcal{A}\times\mathcal{C})$. Since $\mathcal{A}$ is a closed and bounded subset of $\mathbb{R}$, their lemma shows that tangential Hadamard directional differentiability holds for any fixed $c\in\mathcal{C}$. We show that this holds uniformly in $c\in\mathcal{C}$ as well. First, by their equation (B.1), we note that for some $t_{n}\searrow 0$,

[TABLE]

Second, by their equations leading to (B.3)

[TABLE]

by uniform continuity of $h(a,c)$ in $a$ and $c$, which follows from the continuity of $h$ on its compact support $\mathcal{A}\times\mathcal{C}$. Finally, combining equations (19) and (20) as in equation (B.4) from Fang and Santos (2019), it follows that

[TABLE]

which completes the proof. ∎

Proof of lemma 4.

We begin by showing that the first component in equation (11) converges to a tight random element of $\ell^{\infty}(\mathcal{C}\times[0,1])$. Fix $c$ and $w$ and define

[TABLE]

by

[TABLE]

As in the proof of lemma 1, the four mappings $(\delta_{1},\delta_{2},\delta_{3},\delta_{4})$ when considered from $\ell^{\infty}(\mathbb{R}\times\{0,1\}\times\operatorname*{supp}(W))\times\ell^{\infty}(\{0,1\}\times\operatorname*{supp}(W))\times\ell^{\infty}(\operatorname*{supp}(W))$ to $\ell^{\infty}(\mathbb{R}\times\{0,1\}\times\operatorname*{supp}(W))$ are all Hadamard differentiable when evaluated at $\theta_{0}$.

We can write

[TABLE]

By linearity $(\delta_{j}-\delta_{k})(\theta)$ is Hadamard differentiable at $\theta_{0}$ for $j=1,2$ and $k=3,4$. By the chain rule (lemma 6) and lemma 8, the mappings

[TABLE]

are Hadamard directionally differentiable at $\theta_{0}$ for $j=1,2$ and $k=3,4$. Finally, the maximum operator over five arguments is Hadamard directionally differentiable, and by another application of the chain rule, $\phi_{4}$ is Hadamard directionally differentiable for fixed $c$ and $w$. Uniformity over $c\in\mathcal{C}$ and $w\in\operatorname*{supp}(W)$ is obtained from considering the vector of Hadamard directional derivatives for all $c\in\mathcal{C}$ and $w\in\operatorname*{supp}(W)$.

By lemma 3, the mapping $(F_{Y\mid X,W}(\cdot\mid\cdot,\cdot),p_{(\cdot|\cdot)},q_{(\cdot)})\mapsto\underline{P}(\cdot\mid\cdot)$ is Hadamard directionally differentiable. Linearity of the Hadamard directional derivative operator yields that the mapping $(F_{Y\mid X,W}(\cdot\mid\cdot,\cdot),p_{(\cdot|\cdot)},q_{(\cdot)})\mapsto\underline{\text{CDTE}}(z,\cdot,\cdot\mid\cdot)$ is Hadamard directionally differentiable.

Since

[TABLE]

the infimum operator is Hadamard directionally differentiable. As in the proof of lemma 1, the minimum operator is Hadamard directionally differentiable. Following lemma 3, the mapping $(F_{Y\mid X,W}(\cdot\mid\cdot,\cdot),p_{(\cdot|\cdot)},q_{(\cdot)})\mapsto\overline{P}(\cdot\mid\cdot)$ is Hadamard directionally differentiable. A similar argument as above implies the mapping $(F_{Y\mid X,W}(\cdot\mid\cdot,\cdot),p_{(\cdot|\cdot)},q_{(\cdot)})\mapsto\overline{\text{CDTE}}(z,\cdot,\cdot\mid\cdot)$ is Hadamard directionally differentiable.

Combining these results with lemma 5 allows us to conclude that

[TABLE]

a tight random element of $\ell^{\infty}(\mathcal{C}\times[0,1]\times\operatorname*{supp}(W),\mathbb{R}^{2})$ with continuous paths.

Finally, to see that equation (11) holds, consider the lower bound estimator. We have

[TABLE]

A similar derivation holds for the upper bound estimator. ∎

Proof of theorem 2.

By lemmas 3 and 5, the numerator of equation (13) converges uniformly over $c\in\mathcal{C}$. By lemmas 3 and 4, the denominator also converges uniformly over $c\in\mathcal{C}$. By the delta method,

[TABLE]

where

[TABLE]

and

[TABLE]

Here $\tilde{\mathbf{Z}}(c)$ is a random element of $\ell^{\infty}(\mathcal{C})$ by lemmas 3 and 4. Therefore,

[TABLE]

converges to a random element in $\ell^{\infty}(\mathcal{C}\times\mathcal{P})$.

As discussed in the proof of lemma 1, the maximum and minimum operators in equation (12) are Hadamard directionally differentiable. By lemma 6, their composition is Hadamard directionally differentiable. Therefore, by the delta method for Hadamard directionally differentiable functions, $\sqrt{N}(\widehat{\text{BF}}(c,\underline{p})-\text{BF}(c,\underline{p}))$ converges in process as in the statement of the theorem. ∎

Lemma 9**.**

Let $h:A\rightarrow\mathbb{R}$ where $A\subseteq\mathbb{R}$. Let $F(h)=\sup_{x\in A}h(x)$. Let $\|\cdot\|_{\infty}$ denote the sup-norm $\|h\|_{\infty}=\sup_{x\in A}|h(x)|$. Then $F$ is Lipschitz continuous with respect to the sup-norm $\|\cdot\|_{\infty}$ and has Lipschitz constant equal to one.

Proof of lemma 9.

For functions $h$ and $h^{\prime}$,

[TABLE]

By a symmetric argument,

[TABLE]

Therefore $|F(h)-F(h^{\prime})|\leq\|h-h^{\prime}\|_{\infty}$. ∎

Proof of proposition 2.

Hadamard directional differentiability of $\phi$ follows from the chain rule (lemma 6) and from the proof of theorem 2, since the breakdown frontier is a Hadamard directionally differentiable mapping of $\overline{P}(\cdot)=\mathbb{E}[\overline{P}(\cdot\mid W)]$ and $\overline{\text{DTE}}(z,\cdot,\cdot)$, which are themselves Hadamard directionally differentiable mappings of $\theta_{0}$.

Lemma 5 combined with theorem 3.6.1 of van der Vaart and Wellner (1996) implies consistency of the nonparametric bootstrap for our underlying parameters: $\mathbf{Z}_{N}^{*}=\sqrt{N}(\widehat{\theta}^{*}-\widehat{\theta})\overset{P}{\rightsquigarrow}\mathbf{Z}_{1}$. By this result, $\varepsilon_{N}\rightarrow 0$, $\sqrt{N}\varepsilon_{N}\rightarrow\infty$, and theorem 3.1 in Hong and Li (2018), equation (15) holds.

By $1/\sigma(c)$ being uniformly bounded, we have that

[TABLE]

By lemma 9, the $\sup$ operator is Lipschitz with Lipschitz constant equal to 1. Therefore, by proposition 10.7 on page 189 of Kosorok (2008), we can apply a continuous mapping theorem to get

[TABLE]

The rest of the proof follows from corollary 3.2 of Fang and Santos (2015). ∎

Proof of corollary 1.

This follows immediately from proposition 2 and lemma 10 below. ∎

Lemma 10**.**

Let $\overline{C}>0$ and $\mathcal{C}=\{c_{1},\ldots,c_{J}\}\subseteq[0,\overline{C}]$ be a finite grid of points. Let $f:[0,\overline{C}]\rightarrow\mathbb{R}_{+}$ be a nonincreasing function. Let $\widehat{\text{LB}}(\cdot)$ be an asymptotically exact uniform lower $1-\alpha$ confidence band for $f$ on the grid points:

[TABLE]

Define $\widetilde{\text{LB}}:[0,\overline{C}]\rightarrow\mathbb{R}_{+}$ by

[TABLE]

Then $\widetilde{\text{LB}}(\cdot)$ is an asymptotically exact uniform lower $1-\alpha$ confidence band on $[0,\overline{C}]$:

[TABLE]

Proof of lemma 10.

Define the events

[TABLE]

and

[TABLE]

$A$ immediately implies $B$. Since $f$ is nonincreasing, $B$ implies $A$. Thus

[TABLE]

The first line follows by definition of $\widetilde{\text{LB}}$ and since $f$ is nonnegative. Taking limits as $N\rightarrow\infty$ yields

[TABLE]

where the last equality follows from the validity of the band $\widehat{\text{LB}}(\cdot)$ on $\mathcal{C}$. ∎