Counterfactual Analysis under Partial Identification Using Locally Robust Refinement

TL;DR

This paper introduces a method to identify more robust counterfactual predictions within partially identified models, especially in the presence of multiple equilibria, by focusing on locally robust refinements.

Contribution

It proposes a new approach to select desirable parameter values for counterfactual analysis that are robust to local perturbations within the identified set.

Findings

Robust counterfactual predictions can be identified within the identified set.

The method simplifies implementation through a new representation.

Application to moment inequality models and top-coded data demonstrates practicality.

Abstract

Structural models that admit multiple reduced forms, such as game-theoretic models with multiple equilibria, pose challenges in practice, especially when parameters are set-identified and the identified set is large. In such cases, researchers often choose to focus on a particular subset of equilibria for counterfactual analysis, but this choice can be hard to justify. This paper shows that some parameter values can be more "desirable" than others for counterfactual analysis, even if they are empirically equivalent given the data. In particular, within the identified set, some counterfactual predictions can exhibit more robustness than others, against local perturbations of the reduced forms (e.g. the equilibrium selection rule). We provide a representation of this subset which can be used to simplify the implementation. We illustrate our message using moment inequality models, and provide an empirical application based on a model with top-coded data.

Tables

Table 1. Table 1 . The Empirical Coverage Probability and Average Length of Confidence Intervals for ψ ​ ( β 0 , γ 0 , π 0 ) 𝜓 subscript 𝛽 0 subscript 𝛾 0 subscript 𝜋 0 \psi(\beta_{0},\gamma_{0},\pi_{0}) at 95% Nominal Level.

		Specification 1		Specification 2
$\psi$		Least Favorable	RSW	Least Favorable	RSW
Identified Set	$n=500$	1	0.996	1	1
LRR	$n=500$	0.95	0.95	1	1

Table 2. Table 2 . Confidence Intervals for ψ 𝜓 \psi (ATE), Parameter of Interest

	5% top-coding	10% top-coding
Identified Set - LF	[0.023, 0.245]	[0, 0.245]
Identified Set - RSW	[0.041, 0.245]	[0, 0.245]
LRR - LF	[0.034, 0.110]	[0, 0.091]
LRR - RSW	[0.041, 0.110]	[0, 0.091]

Notation	Description	Place of Def.
$B_P$	: Identified set for $\beta$, the parameter of interest, given $P\in 𝐏$
${\beta }_0$	: The parameter of interest.
$C_{1-\alpha }^{\mathrm{𝖫𝖱𝖱}}$	: A bootstrap confidence region for ${\beta }_0$ under the LRR hypothesis.
	(based on the least favorable configuration)	(4.10)
${\tilde{C}}_{1-\alpha }^{\mathrm{𝖫𝖱𝖱}}$	: A bootstrap confidence region for ${\beta }_0$ under the LRR hypothesis.
	(based on the approach of Romano/Shaikh/Wolf:14:Eca)	(4.16)
${\hat{c}}_{1-\alpha }\left(\beta \right)$	: The $1-\alpha$ quantile bootstrap critical value.	Above (4.10)
${\tilde{c}}_{1-\alpha +{\alpha }_1}\left(\beta \right)$	: The $1-\alpha +{\alpha }_1$ quantile bootstrap critical value.	Above (4.16)
$F$	: The distribution of $W=(X,\epsilon )$ in the DGP for the data.
$\tilde{F}$	: The counterfactual distribution of $W=(X,\epsilon )$.
$G(\cdot |w)$	: The conditional distribution of $\eta$ given $W=w$, as the reduced-
	form selection rule.
${𝒢}_n$	: The $\sigma$-field generated by the observations ${\left\{{(Y_i,X_i^{\prime })}^{\prime }\right\}}_{i=1}^n$.
$\Gamma \left(\beta \right)$	: Identified set for ${\gamma }_0$, given $\beta$ and $P\in 𝐏$	(3.8)
${\Gamma }_{\kappa }^{\mathrm{𝖫𝖱𝖱}}\left(\beta \right)$	: The $\kappa$-LRR set.	(3.12)
${\hat{\Gamma }}_{\kappa }\left(\beta \right)$	: The estimated version of ${\Gamma }_P\left(\beta \right)$.	(4.5)
${\hat{\Gamma }}_{\kappa }^{\mathrm{𝖫𝖱𝖱}}\left(\beta \right)$	: The estimated upper LRR set	(4.11)
${\gamma }_0$	: The nuisance parameter.
$h_n,{\overline{h}}_n^M,{\overline{h}}_n^M$	: Functionals used to define various statistics.	(A.33), (A.34)
$m_{j,P}(\beta ,\gamma )$	: The expected moment function in the moment inequality model	(A.2)
${\overline{m}}_j(\beta ,\gamma )$	: The average moment function in the moment inequality model	(4.2)
${\hat{\nu }}_{n,j,P}(\beta ,\gamma )$	: Empirical processes normalized by ${\hat{\sigma }}_j(\beta ,\gamma )$	(A.24)
${\tilde{\nu }}_{n,j,P}(\beta ,\gamma )$	: Empirical processes normalized by ${\sigma }_{j,P}(\beta ,\gamma )$	(A.24)
${\hat{\nu }}_{n,j}^{\ast }(\beta ,\gamma )$	: Bootstrap empirical processes normalized by ${\hat{\sigma }}_j(\beta ,\gamma )$	(A.25)
${\tilde{\nu }}_{n,j}^{\ast }(\beta ,\gamma )$	: Bootstrap empirical processes normalized by ${\sigma }_{j,P}(\beta ,\gamma )$	(A.25)
$Q_P(\beta ,\gamma )$	: The population obj. function used to set-identify $({\beta }_0,{\gamma }_0)$	(A.31)
$\hat{Q}(\beta ,\gamma )$	: The sample obj. function from the moment inequality model.	(4.4)
${\hat{Q}}^{\ast }(\beta ,\gamma )$	: The bootstrap sample objective function.
	(based on the least favorable configuration.)	(4.7)
${\tilde{Q}}^{\ast }(\beta ,\gamma )$	: The bootstrap sample objective function.
	(based on the approach of Romano/Shaikh/Wolf:14:Eca)	(4.14)
$Q^{\mathrm{𝖫𝖱𝖱}}(\beta ,\gamma )$	: The population criterion for LRR.	(3.10)
$T(\beta ,\gamma )$	: The test statistic.	(4.6)
${\tilde{T}}^{\ast }(\beta ,\gamma )$	: The bootstrap test statistic.
	(based on the approach of Romano/Shaikh/Wolf:14:Eca)	(4.15)
${\Theta }_P$	: Identified set for $({\beta }_0,{\gamma }_0)$ given $P\in 𝐏$	Section 3.1
${\Theta }_P^{\mathrm{𝖫𝖱𝖱}}$	: The set of $({\beta }_0,{\gamma }_0)\in {\Theta }_P$ under the LRR hypothesis.	(3.13)

Equations

$$y={\rho }_{\beta ,\gamma }(x,\epsilon ;\eta ),$$

$${\text{ASF}}_{\beta ,\gamma ,G}(x)=\int \int {\rho }_{\beta ,\gamma }(x,\epsilon ;\eta )dG(\eta |x,\epsilon )d{F}_{\epsilon }(\epsilon ),$$

$$\underset{G^{\prime }\in B(G;\delta )}{\sup }\left|\int {\text{ASF}}_{\beta ,\gamma ,G}(x)d{\tilde{F}}_X(x)-\int {\text{ASF}}_{\beta ,\gamma ,G^{\prime }}(x)d{\tilde{F}}_X(x)\right|,$$

$$Y=\rho (W;\eta ),$$

$$C(\rho ,\tilde{F},G)=\int \rho (w;\eta )dG(\eta |w)d\tilde{F}(w).$$

$$\{C(\rho ,\tilde{F},G):G\in \mathcal{G},\rho \in \mathcal{R}\}.$$

$$\mathbb{W}(a)=\left\{w:u_i(a_i,a_{-i},w)\ge \underset{a^{\prime }\in A}{\max }{u}_i(a^{\prime },a_{-i},w),\forall i=1,...,N\right\}.$$

$$\mathcal{A}(w)=\left\{a\in A^N:w\in \mathbb{W}(a)\right\}$$

$$Y_i={\rho }_i(W;\eta ),$$

$$\rho (w;\eta )=\sum _{a\in \mathcal{A}(w)}1\{\eta =a\}a.$$

$$\displaystyle Y=\left\{\begin{array}[]{ll}g_{1}\left(\beta_{0}D+X_{1}^{\prime}\gamma_{0}\right)+\varepsilon,&\text{ if }\eta=1,\text{ and }\\ g_{2}\left(\beta_{0}D+X_{1}^{\prime}\gamma_{0}\right)+\varepsilon,&\text{ if }\eta=0,\end{array}\right.$$

$${\Theta }_P=\left\{({\beta }_0,{\gamma }_0):\mathbf{E}[{\tilde{Y}}_1|X_1,D]\le \alpha (X_1,D;{\beta }_0,{\gamma }_0,\pi )\le \mathbf{E}[{\tilde{Y}}_2|X_1,D],\text{a.e.}\text{ for some }\pi \in \Pi \right\},$$

$$\alpha (X_1,D;{\beta }_0,{\gamma }_0,\pi )=g_1\left({\beta }_{0}D+X_1^{\prime }{\gamma }_0\right)\pi (X_1,D)+g_2\left({\beta }_{0}D+X_1^{\prime }{\gamma }_0\right)(1-\pi (X_1,D)).$$

$$Y=\rho (X,\epsilon ;\eta ),$$

$$\rho (X,\epsilon ;\eta )=g_1\left({\beta }_{0}D+X_1^{\prime }{\gamma }_0\right)\eta +g_2\left({\beta }_{0}D+X_1^{\prime }{\gamma }_0\right)(1-\eta )+\epsilon .$$

$$Y={\rho }_{{\beta }_0,{\gamma }_0}(W;\eta ),$$

$${\text{ASO}}_{{\beta }_0,{\gamma }_0}(G)=\int \int {\rho }_{{\beta }_0,{\gamma }_0}(w;\eta )dG(\eta |w)d\tilde{F}(w).$$

$$S_{\beta ,\gamma }(G;K)=\underset{G^{\prime }\in \mathcal{G}:{\delta }_{\mathcal{G}}(G^{\prime },G)\le K}{\sup }\frac{|{\text{ASO}}_{\beta ,\gamma }(G^{\prime })-{\text{ASO}}_{\beta ,\gamma }(G)|}{{\delta }_{\mathcal{G}}(G^{\prime },G)},$$

$${\Gamma }^{\mathsf{LRR}}(\beta )={\text{argmin}}_{\gamma :(\beta ,\gamma )\in {\Theta }_P}{S}_{\beta ,\gamma }(G;K),$$

$${\Theta }_P^{\mathsf{LRR}}=\{(\beta ,\gamma )\in {\Theta }_P:\gamma \in {\Gamma }^{\mathsf{LRR}}(\beta )\}.$$

$$({\beta }_0,{\gamma }_0)\in {\Theta }_P^{\mathsf{LRR}}.$$

$${\delta }_{\mathcal{G}}(G^{\prime },G)=\sqrt{\int \int {\left(\frac{d{G}^{\prime }}{d\mu }-\frac{dG}{d\mu }\right)}^2(\eta |w)d\mu (\eta )d{\tilde{F}}_P(w)}.$$

$$\Gamma (\beta )=\{\gamma \in \Gamma :(\beta ,\gamma )\in {\Theta }_P\},$$

$${\Gamma }_{\kappa }^{\mathsf{LRR}}(\beta )=\left\{\gamma \in \Gamma (\beta ):S_{\beta ,\gamma }^2(G;K)\le S_{\beta ,\tilde{\gamma }}^2(G;K)+\kappa ,\forall \tilde{\gamma }\in \Gamma (\beta )\right\},$$

$${\Delta }_{\beta ,\gamma }(w;\eta )={\rho }_{\beta ,\gamma }(w;\eta )-\int {\rho }_{\beta ,\gamma }(w;\eta )d\mu (\eta ).$$

$$Q^{\mathsf{LRR}}(\beta ,\gamma )=\int \int {\Delta }_{\beta ,\gamma }^2(w;\eta )d\mu (\eta )d\tilde{F}(w).$$

$$S_{\beta ,\gamma }(G;K)=\sqrt{Q^{\mathsf{LRR}}(\beta ,\gamma )},$$

$${\Gamma }_{\kappa }^{\mathsf{LRR}}(\beta )=\left\{\gamma \in \Gamma (\beta ):Q^{\mathsf{LRR}}(\beta ,\gamma )\le Q^{\mathsf{LRR}}(\beta ,\bar{\gamma })+\kappa ,\text{ for all }\bar{\gamma }\in \Gamma (\beta )\right\}.$$

$${\Theta }_{\kappa ,P}^{\mathsf{LRR}}=\left\{(\beta ,\gamma )\in {\Theta }_P:\gamma \in {\Gamma }_{\kappa }^{\mathsf{LRR}}(\beta )\right\}.$$

$${\Psi }_{\kappa }^{\mathsf{LRR}}=\left\{\psi (\beta ,\gamma ):(\beta ,\gamma )\in {\Theta }_{\kappa ,P}^{\mathsf{LRR}}\right\}.$$

Full text

Key words: Counterfactual Analysis, Multiple Equilibria, Equilibrium Selection Rule, Locally Robust Refinement

JEL Classification: C30, C57

We thank three anonymous referees for their valuable comments. Corresponding author: Kyungchul Song, kysong@mail.ubc.ca, Vancouver School of Economics, University of British Columbia, 6000 Iona Drive, Vancouver, BC, V6t 1Z4, Canada

1. Introduction

Economists often use a structural model to analyze the effect of a policy that has never been implemented, such as potential mergers or changes to legislation. For such analysis, it is crucial to have a plausible specification of the causal relationship between the policy variable and the outcome variable of interest. However, a plausible specification can come short of determining the causal relationship uniquely. Prominent examples are found in game theoretic models, where the presence of multiple equilibria admits multiple reduced forms. (Tamer:03:ReStud called models with multiple reduced forms incomplete.)111In this paper, a reduced form refers to a system of equations where the variables on the right hand side are all external variables (i.e. inputs) and the variables on the left hand side are all internal variables (i.e. generated through the system of equations using the inputs) in the sense of Heckman:10:QJE (p.56). The term “reduced form” in this sense appears in econometrics at least as early as in Koopmans:49:Eca. The multiplicity of reduced forms is often the source of set-identification of structural parameters.

When a parameter is set-identified, all the values in the identified set are empirically equivalent, in the sense that the observed data cannot be used to distinguish between the values. The identified set represents the empirical content of the model for the parameter (i.e., the amount of information on the value of the parameter that can be extracted from data through the model). However, when a parameter is set-identified with a large identified set, the identified set can be of little use for counterfactual analysis in practice. It is not unusual that predictions from counterfactual analysis based on different equilibria yield self-contradictory results (Aguirregabiria/Nevo:12:WP, p.111). In this situation, just as in the calibration approach in macroeconomics, the researcher typically uses additional constraints on the parameter values or chooses a specific equilibrium to obtain a meaningful analysis.222For example, in an entry model, Jia:08:Eca focuses on the equilibrium that is most profitable to one of the firms, and Ackerberg/Gowrisankaran:06:RAND use an equilibrium selection rule which randomizes between a Pareto-best equilibrium and a Pareto-worst equilibrium with an unknown probability. Another example is Roberts/Sweeting:13:AER in the context of auctions, who consider the particular equilibrium with the lowest signal for agents in the model so they enter the auction. In other words, the researcher considers a refinement of the identified set, by introducing additional restrictions. However, these restrictions are often only weakly motivated.

In our view, a desirable approach in this situation is to distinguish between the identification step and the refinement step, and lay out transparently what criterion (or motivation) is used for the refinement step. There is a small body of literature that explores this refinement step. Song:14:ET analyzed the problem of choosing a point from the identified interval based on the local asymptotic minimax regret criterion. Aryal/Kim:13:JBES considered an incomplete English auction model where the seller is ambiguity averse, and employing the $\Gamma$-maximin expected utility framework of Gilboa/Schmeidler:89:JME, suggested a point decision on the optimal reserve price. Jun/Pinkse:17:WPa considered an incomplete English auction set-up similar to that in Haile/Tamer:03:JPE and proposed a point decision on the optimal reserve price using a maximum entropy method. Jun/Pinkse:19:JOE studied a discrete two-player complete information game and established sharp identification of counterfactual predictions of the game. Furthermore, they investigated point decisions on the probability distributions through a maximum entropy method and what they called a Dirichlet approach, and compared the two methods.

The main goal of this paper is to draw attention to the fact that some values in the identified set can give counterfactual predictions that are more robust to the change of certain aspects of model specification than others. Thus, it is reasonable to give priority to those values that produce most “reliable” counterfactual predictions. Generally speaking, predictions that are sensitive to a change of the aspects of the model that the researcher is not sure about cannot be reliable. Given that the researcher is often least sure about how a reduced form is determined among multiple reduced forms, we consider a subset of the identified set which produces counterfactual predictions that are more robust to local perturbations of the reduced forms (e.g. local perturbations of the equilibrium selection rule in a game setting) than others.

More specifically, consider a causal relationship between endogenous variable $y$ and exogenous variables $x,\varepsilon$:

[TABLE]

where $\beta$ is the parameter of interest, $\gamma$ a nuisance parameter, and $\eta$ is a random variable with distribution $G$. Without observing $\eta$ or knowing $G$, the model admits multiple reduced forms. For example, in the case of games with multiple equilibria, $G$ plays the role of the equilibrium selection rule.333As shown in the paper, our approach applies to causally incomplete models other than game-theoretic models, such as models involving top-coded data.

Suppose that the counterfactual setting of interest involves a counterfactual distribution $\tilde{F}_{X}$ for $X$. Consider the average structural function (ASF) (Blundell/Powell:04:ReStud):

[TABLE]

where $F_{\varepsilon}$ denotes the distribution of $\varepsilon$, and $G(\cdot|x,\varepsilon)$ the conditional distribution of $\eta$ given $x$ and $\varepsilon$. Then our proposal is that we choose $\gamma$ that minimizes

[TABLE]

where $B(G;\delta)$ denotes a $\delta$-neighborhood of $G$ for some small $\delta>0$. (As we show later, given $\beta$, the set of $\gamma$’s that minimize (1.2) does not depend on the choice of $\delta$ or $G$.) We call the resulting set of values $(\beta,\gamma)$ in the identified set the Locally Robust Refinement (LRR). In other words, our refinement is motivated by the desirability of a stable behavior in the counterfactual predictions when the structural parameters are partially identified.

In many set-ups, the LRR is a substantially smaller set than the identified set. Furthermore, the LRR is often very intuitive. For example, as we show in Section 3, the LRR in an entry game picks nuisance parameter values that minimize the region of multiple equilibria. In the example of a switching regression model with interval data, it picks nuisance parameters that minimize a difference across models in switching regression.

However, unlike the identified set, the size of the LRR does not indicate the empirical content of the model. Rather the LRR gives parameter values which produce counterfactual predictions that are most robust to the change of the reduced forms among their multiple forms. Due to this robustness property, one may want to pay more attention to those parameter values than others when conducting policy exercises, even if all the values in the identified set are observationally equivalent.

The formulation in (1.2) gives the impression that the LRR involves additional optimization over an infinite dimensional space in estimation, adding to computational cost. Using the linearity of $\textsf{ASF}_{\beta,\gamma,G}(x)$ in $G$ and Hilbert space geometry, this paper gives a simple characterization of the LRR so that focusing on the LRR does not cause extra computational cost. In fact, due to the additional constraints that come from the LRR, the computational cost is substantially reduced from a benchmark case without using the LRR. Focusing on moment inequality models, this paper presents a method of bootstrap inference on the LRR from the identified set and shows its uniform asymptotic validity. We provide results from Monte Carlo simulations, and illustrate the LRR through an empirical application on top-coded data. These exercises illustrate that the LRR’s predictions are indeed more stable than those with the identified set and that they may affect policymaking in applied contexts (i.e. policies suggested by the identified set could be driven by parameter values that are not robust, and are not part of the LRR).

There has long been interest in studying model sensitivity and robustness to misspecifications in econometric models. Recent contributions in this line include Andrews/Gentzkow/Shapiro:18:NBER, Armstrong/Kolesar:20:QE and Bonhomme/Weidner:20:WP who, among many others, proposed a method of measuring local misspecification and performing inference that is robust to local misspecifications. In the context of instrumental variable models, Nevo/Rosen:12:ReStat permitted the instrumental variable to be correlated with the error term and presented bounds for the parameter of interest under an assumption on the correlation between the instrumental variable and the error term. Oster:19:JBES proposed a method of extracting information on the omitted variable bias from data in linear regression models. Masten/Poirier:18:Ecma provided an identification analysis of various treatment effect parameters from treatment effect models where the conditional independence restriction is weakened to a partial dependence condition.444Robustness to misspecification has also been considered in the macroeconomics literature (e.g. in optimal control theory as in Hansen/Sargent:01:AER, see Hansen/Marinacci:16:SS as well), where one can consider perturbations of the decision maker’s model, but less emphasis is placed on the identification and inference problems in these models. Finally, Andrews/Gentzkow/Shapiro:17:QJE focus on the case where the model is point-identified, and present a statistic on how estimates vary when a model’s moments changes. Christensen/Connault:19:WP proposed a computationally tractable method of performing inference on counterfactual quantities that arise in structural models, while permitting a global misspecification of the model. Li:19:WP uses a semiparametric restriction (instead of parametric ones) to generate bounds for counterfactuals of interest in moment inequality models.

All these authors are concerned about robustness of inference to misspecifications. Their robust inference often leads to a larger confidence set, and a wider range of policy predictions. In contrast, we consider settings where the identified set is already too large to be useful for a meaningful policy analysis. This can happen even if the model is correctly specified (see Aguirregabiria/Nevo:12:WP for a discussion, particularly in the context of multiple equilibria). Instead of enlarging an existing identified set by “robustifying” it as above, we explore a rationale for focusing on a subset of the identified set for policy analysis. Our criterion is based on the stability of counterfactual predictions as we perturb the reduced-form selection rules. One can relate our guiding philosophy to the literature of equilibrium refinements in game theory, where one focuses on a subset of the set of Nash equilibria based on its stability against various perturbations of the game. (See Chapter 5 of Myerson:91:GameTheory.) Then the idea of local robustness in the paper, i.e., the stability of reduced forms against local perturbations of reduced form selection rules is related to Fudenberg/Kreps/Levine:88:JET who considered robustification of the set of strict Nash equilibria against local perturbations of the games.

The paper is organized as follows. The next section gives a general set-up of a structural model and presents examples that fit our framework. Section 3 presents the approach of LRR and provides its characterization that is useful for implementation in practice. Focusing on generic moment inequality models, Section 4 proposes inference for the LRR and presents results from Monte Carlo simulations. An empirical application is presented in Section 5. Section 6 concludes. In the Appendix, we provide some mathematical proofs. The Supplemental Note to this paper includes additional information on the data for the empirical application, and presents results on the uniform validity of our inference procedure.

2. A Counterfactual Analysis with Multiple Reduced Forms

2.1. Models with Multiple Reduced Forms

Suppose that data $\{(Y_{i}^{\prime},X_{i}^{\prime})^{\prime}\}_{i=1}^{n}$ are drawn from the joint distribution $P$ of $(Y^{\prime},X^{\prime})^{\prime}$ which obeys the following reduced form: for $Y\in\mathbf{R}$ and $W=(X^{\prime},\varepsilon^{\prime})^{\prime}$,

[TABLE]

for some function $\rho$, where $X$ is observed but $\varepsilon$ and $\eta$ are not. For each value $\eta$, we regard the map $\rho(\cdot;\eta)$ as representing a functional-causal relationship between internal variable $Y$ and external variables $W$.555Here we follow the proposal by Heckman:10:QJE (p.56) and use the terms internal variables and external variables. External variables are specified from outside of the structural equations as inputs, and internal variables are generated through the system of equations from the inputs. External variables are not necessarily exogenous variables. When $X$ is correlated with $\varepsilon$, the variable $X$ is called endogenous, although $(X,\varepsilon)$ is jointly given externally as inputs, and thus is a vector of external variables. Here a reduced form refers to a system of equations where all the variables on the left hand side are internal variables and all the variables on the right hand side are external variables. This relationship is fully determined once the value of $\eta$ is realized. Typically, $\varepsilon$ has a structural meaning such as unobserved costs, whereas $\eta$ does not and is used only as an index for a reduced form. For example, in a game-theoretic model, $W=(X^{\prime},\varepsilon^{\prime})^{\prime}$ represents observed and unobserved payoff components and $\eta$ an index for an equilibrium. The response of $Y$ to a change in $W$ is ambiguous unless the value of $\eta$ is fixed. In this sense, there is a multiplicity of reduced forms in (2.1). To describe the generation of observed outcome $Y$, we assume that Nature selects the value of $\eta$ from the conditional distribution $G(\cdot|w)$ given $W=w$. In a game setting, $G$ is often called the equilibrium selection rule. However, the multiplicity of reduced forms can arise in a setting that has nothing to do with a game, such as in the case of switching regressions with interval data which we describe below. Hence we call $G$ the reduced-form selection rule in this paper.

2.2. Counterfactual Analysis Under Partial Identification

We focus on a counterfactual experiment where the distribution $F$ of the external vector $W$ is changed to a counterfactual one $\tilde{F}$. Such a counterfactual experiment includes cases where one “shuts off” the effect of a certain covariate, “fixing” of the covariate to a certain value, or changing the distribution of $\varepsilon$. Suppose that the counterfactual quantity of interest is generically denoted by $C(\rho,\tilde{F},G)$. This quantity could be, for example, the prediction of game outcomes or expected payoffs under a counterfactual scenario:

[TABLE]

Or in other applications, $C(\rho,\tilde{F},G)$ could be the average increase in welfare. Let $\mathcal{G}$ be the space of the reduced-form selection rules $G$ and let $\mathcal{R}$ be the identified set for the function $\rho$. Then the identified set for the counterfactual prediction $C(\rho,\tilde{F},G)$ is given by

[TABLE]

A standard approach to make inference based on this identified set faces multiple challenges in practice. The set of predictions can be of little practical use, when the estimated set is too large. Furthermore, estimation of the identified set itself and statistical inference can be computationally intensive.

Due to these difficulties, it is not unusual that empirical researchers focus on a particular subset of $\mathcal{G}$ (such as focusing on a particular equilibrium) or consider a subset of $\mathcal{R}$ (such as by restricting the admissible values of parameters as in the case of calibration). Such a subset selection approach, while inevitable in practice, is not easy to motivate.

2.3. Examples

For brevity, we consider two examples here to illustrate our set-up: a finite game of complete information and a switching regression with interval data.

2.3.1. Finite Game of Complete Information

Consider the setting of a complete information $N$ player game with finite actions, and let $A$ be the finite action space for each player.666A special case is the two player entry game used by Bresnahan/Reiss:91:JOE and Tamer:03:ReStud, among others. Suppose that the payoff for player $i$ playing $\tilde{a}\in A$ when all other players are playing $a_{-i}=(a_{j})_{j\neq i}$ is given by $u_{i}(\tilde{a},a_{-i},W)$ for a map $u_{i}$, where $W=(X,\varepsilon)$ is a payoff-relevant characteristic vector of player $i$. For each $a\in A^{N}$, we define

[TABLE]

Hence if $w$ belongs to $\mathbb{W}(a)$, $a$ is a Nash equilibrium for the game with $w$, so that the set

[TABLE]

represents the set of equilibria for the game with $w$. Let $\eta$ be a random vector taking values in $A^{N}$ where the conditional distribution $G(\cdot|w)$ of $\eta$ given $w$ has a support in $\mathcal{A}(w)$. Let $Y=(Y_{i})_{i=1}^{N}$ be the observed action profile that comes from a Nash equilibrium $\eta$ of the game after Nature draws $\eta$ from $G(\cdot|w)$. Then, we can write for $i\in N$,

[TABLE]

where $\rho_{i}$ is the $i$-th entry of $\rho$ which is defined as

[TABLE]

The conditional distribution $G(\cdot|w)$ over $\eta$ represents the equilibrium selection rule.

2.3.2. A Switching Regression Model with Interval Outcomes

Consider an interval data set-up similar to one studied by Manski/Tamer:02:ECMA. More specifically, we consider top-coded data with threshold variables $Z_{1}<Z_{2}$, where $Z_{1},Z_{2}$ are observed and common across individuals. We observe an outcome random variable which equals a latent $Y$ if $Y\leq Z_{1}$ and is only known to be within $[Z_{1},Z_{2}]$ otherwise, and a vector of covariates $X_{1}$. For example, $X_{1}$ represents a vector of covariates for each worker such as age, education, experience, gender, and $Y$ is the log of the hourly wage.

It is assumed that the latent $Y$ follows:

[TABLE]

where $D$ denotes the policy variable, and $\eta$ the binary indicator of a regime, and $g_{1}$ and $g_{2}$ are some known functions. As for the error term $\varepsilon$, we assume that $\mathbf{E}[\varepsilon|X_{1},D]=0$. We also assume that $Y\leq Z_{2}$ with probability one. Let $\tilde{Y}_{1}=\tilde{Y}_{2}=Y$ if $Y\leq Z_{1}$, and $\tilde{Y}_{1}=Z_{1}$ and $\tilde{Y}_{2}=Z_{2}$ if $Y>Z_{1}$. The econometrician observes $\tilde{Y}_{1}$ and $\tilde{Y}_{2}$. Let $\pi_{0}(X_{1},D)=P\{\eta=1|X_{1},D\}$ and $\Pi$ be the collection of functions that $\pi_{0}$ is known to belongs to.

It is straightforward that the identified set for $(\beta_{0},\gamma_{0})$ is given by

[TABLE]

where

[TABLE]

Now we can write

[TABLE]

where $X=(X_{1},D)$, and

[TABLE]

The distribution $G$ of $\eta$ plays a role analogous to the equilibrium selection rule in a game-theoretic model. Here it selects from a given set of reduced forms.

3. Locally Robust Refinement

3.1. Overview

Suppose that the reduced-form $\rho(\cdot;\eta)$ in (2.1) is parametrized as follows:

[TABLE]

where $\beta_{0}$ is a parameter of interest taking values in a set $B$ and $\gamma_{0}$ is a nuisance parameter taking values in a set $\Gamma$, and $\eta$ belongs to some known compact set denoted by $H$. Let $\Theta_{P}$ be the identified set for $(\beta_{0},\gamma_{0})$. The counterfactual experiment involves changing the distribution $F$ of $W$ to a counterfactual distribution $\tilde{F}$. Let us consider the counterfactual average structural outcome:

[TABLE]

This is the average value of outcome $Y$ when a counterfactual change is made to the distribution of $W$ while everything else remains the same. Our focus is on a subset of $\Theta_{P}$ at which $\textsf{ASO}_{\beta,\gamma}(G)$ behaves robustly as we perturb the reduced-form selection rule $G$.

More formally, for any $K>0$, we introduce a $K$-sensitivity of $\textsf{ASO}_{\beta,\gamma}(G)$ as we perturb $G$ around in its $K$-neighborhood:

[TABLE]

where $\mathcal{G}$ denotes the collection of $G$’s permitted in the model. The quantity $S_{\beta,\gamma}(G;K)$ measures the maximal rate of change of counterfactual average outcome when $G$ is perturbed within its $K$-neighborhood, where this neighborhood is defined using a metric $\delta_{\mathcal{G}}(G^{\prime},G)$, formally defined below. We search for values of nuisance parameters $\gamma$ which minimize this sensitivity. For each $\beta\in B$, if we let

[TABLE]

we define

[TABLE]

We call the subset $\Theta_{P}^{\mathsf{LRR}}$ the Locally Robust Refinement (LRR) of $\Theta_{P}$. Hence, the LRR can be viewed as a collection of parameters at which the ASO is stable against local misspecifications of $G$. As we will see later in the characterization of the LRR set, the set is independent of the choice of $K$, $G$ and $\mathcal{G}$. The main reason behind this independence is that the functional $\textsf{ASO}_{\beta,\gamma}(G)$ is linear in $G$. (This is analogous to that the slope of a linear function is constant on the domain of the function.)777As long as the true reduced-form selection rule lies in the interior of $\mathcal{G}$, our characterization of the LRR set does not depends on the choice of $\mathcal{G}$. While it is possible to recover some information about the rule $G$ with further specifications, our procedure remains agnostic about $\mathcal{G}$, which is in line with many existing works in the econometrics literature of game-theoretic models - see Ciliberto/Tamer:09:Eca for one such example. Note that, as we do not want our concern of local robustification against the reduced-form selection rules to directly affect the parameter of interest, our approach subjects only the nuisance parameters to the refinement.

The inference on $\beta$ based on the Locally Robust Refinement proceeds under the further restriction as follows.

[TABLE]

We call this restriction the LRR Hypothesis. Once we focus on the LRR set of parameters, this can be used for various exercises of counterfactual analysis, just as one does with the identified set of parameters. Despite the impression that finding the set $\Theta_{P}^{\mathsf{LRR}}$ might be computationally intensive as it involves layers of optimization, it is not, due to our characterization result in the next section.

3.2. Characterization of the Locally Robust Refinement

To derive a characterization of the LRR, let us introduce some geometric structure on the space of reduced-form selection rules $G$. Let $\mathcal{G}$ be the collection of all the conditional distributions $G$ (of $\eta$ given $W$) which are dominated by the uniform distribution $\mu$ over $H$, and $w\in\mathcal{W}$, with $\mathcal{W}$ denoting the set $W$ takes values from. Let us introduce the following metric on $\mathcal{G}$: for $G,G^{\prime}\in\mathcal{G}$,

[TABLE]

Let us define

[TABLE]

i.e., the set of nuisance parameters that are admitted in the counterfactual analysis together with given $\beta$. For $\kappa\geq 0$, we define the $\kappa$-LRR set for $\gamma$ corresponding to $\beta$ as

[TABLE]

where $S_{\beta,\gamma}(G;K)$ is defined in (3.3). The set $\Gamma^{\mathsf{LRR}}(\beta)$ defined in (3.4) is a special case of $\Gamma_{\kappa}^{\mathsf{LRR}}(\beta)$ with $\kappa=0$. However, as we discuss in Section 4.1, it is convenient to use the LRR set with a small positive number $\kappa>0$ which allows us to develop uniform inference. As shown below, this set does not depend on the choice of $K>0$ or that of $G\in\mathcal{G}$.

We now provide a useful characterization of the set $\Gamma_{\kappa}^{\mathsf{LRR}}(\beta)$. For this, we define

[TABLE]

The function $\Delta_{\beta,\gamma}(w;\eta)$ is a mean-deviation form of the structural function $\rho_{\beta,\gamma}(w;\eta)$. This function plays a central role in the characterization result. Let us make the following assumption that guarantees that the average variability of this mean-deviation is well defined.

Assumption 3.1**.**

For each $\beta\in B$ and $\gamma\in\Gamma(\beta)$, $Q^{\mathsf{LRR}}(\beta,\gamma)<\infty$, where

[TABLE]

In applications where we can compute $\rho_{\beta,\gamma}(w;\eta)$ for each value of $\eta$, we can evaluate $Q^{\mathsf{LRR}}(\beta,\gamma)$ easily. (See Section 3.3 below for examples.) The following theorem shows that the set $\Gamma_{\kappa}^{\mathsf{LRR}}(\beta)$ is fully characterized through the function $Q^{\mathsf{LRR}}(\beta,\gamma)$.

Theorem 3.1**.**

Suppose that Assumption 3.1 holds. Then for each $\beta\in B$ and $\gamma\in\Gamma(\beta)$,

[TABLE]

for all $G\in\mathcal{G}$ and all $K>0$.

Thus the $\kappa$-LRR of $\gamma_{0}$ can be rewritten as

[TABLE]

The $\kappa$-LRR of the nuisance parameter vector $\gamma_{0}$ is the set of $\gamma$’s which minimize the average variability of $\rho_{\beta,\gamma}(w;\eta)$ (around its mean) as we perturb $\eta$. Thus, the $\kappa$-LRR for $(\beta_{0},\gamma_{0})$ is given by

[TABLE]

Sometimes, the target parameter may not be a parameter $\beta$ that determines the reduced form $\rho_{\beta,\gamma}$ but some aggregate quantity such as Average Treatment Effects or aggregate welfare. Suppose that such a target parameter is identified up to $\beta_{0}$ and $\gamma_{0}$, so that we can write it as, say $\psi(\beta_{0},\gamma_{0})$. (We give an example in Section 3.3.2 below.) Then, the LRR set for $\psi(\beta_{0},\gamma_{0})$ is given by

[TABLE]

To build familiarity with our characterization results for the LRR and show how it can be used in practice, we now derive $Q^{LRR}$ for two applied examples.

3.3. Examples

3.3.1. Entry Game with Nash Equilibria

Consider the setting of a 2x2 complete information entry games, such as the empirical setting in Bresnahan/Reiss:91:JOE and Ciliberto/Tamer:09:Eca, where there are two firms, $i=1,2$, deciding whether to enter a market or not. The entry decisions of the firms can then be formulated as

[TABLE]

where $\beta_{1,0}D_{2}+X^{\prime}\gamma_{1,0}-\varepsilon_{1}$ represents the payoff difference for firm $1$ between entering and not entering the market, $D_{i}=1$ represents the entry decision by firm $i$ and $D_{i}=0$ the decision not to enter by firm $i$. The random vectors $X$ and $\varepsilon$ represent respectively the observed and unobserved characteristics of firms $1$ and $2$ combined. The coefficients $\beta_{1,0}$ and $\beta_{2,0}$ capture strategic interactions between firms and they are our coefficients of interest, whereas the coefficients $\gamma_{1,0}$ and $\gamma_{2,0}$ measure the roles of firm specific and market specific characteristics and, as such, are nuisance parameters.

[INSERT FIGURE 1 HERE]

In order to find a reduced form as in (2.1), we first put

[TABLE]

where $Y\in\mathcal{Y}=\{(0,0),(1,1),(0,1),(1,0)\}$. Define (with $\beta_{0}=(\beta_{1,0}^{\prime},\beta_{2,0}^{\prime})^{\prime}$ and $\gamma_{0}=(\gamma_{1,0}^{\prime},\gamma_{2,0}^{\prime})^{\prime}$)

[TABLE]

These regions are represented on the left panel of Figure 1.

Let $\rho_{\beta,\gamma}(x,\varepsilon;\eta)=(\rho_{\beta,\gamma,1}(x,\varepsilon;\eta),\rho_{\beta,\gamma,2}(x,\varepsilon;\eta))$, where for a random variable $\eta\in\{0,1\}$,

[TABLE]

where $A_{3\cap 4,\beta,\gamma}(x)=A_{3,\beta,\gamma}(x)\cap A_{4,\beta,\gamma}(x)$, $A_{4\setminus 3,\beta,\gamma}(x)=A_{4,\beta_{0},\gamma_{0}}(x)\setminus A_{3,\beta_{0},\gamma_{0}}(x)$, and $A_{3\setminus 4,\beta,\gamma}(x)=A_{3,\beta_{0},\gamma_{0}}(x)\setminus A_{4,\beta_{0},\gamma_{0}}(x)$. Then we can write reduced forms for $Y_{i}$, for firm $i=1,2$ as

[TABLE]

Let the distribution $G$ of $\eta$ be a discrete distribution with support $\{0,1\}$, and take $\mu$ to be a Bernoulli random variable with a mean $1/2$.888This example focuses on pure-strategy Nash equilibria following the empirical literature on entry-exit games (e.g. Tamer:03:ReStud, Jia:08:Eca, Ciliberto/Tamer:09:Eca, among many others). It follows that for $i=1,2$,

[TABLE]

Hence, by Theorem 3.1, the LRR set based on the ASO of $Y_{1}$ is the same as that based on the ASO of $Y_{2}$. Writing the mean-deviated reduced form on the left hand side of (3.19) as $\Delta_{\beta,\gamma}(x,\varepsilon;\eta)$, we find that

[TABLE]

Plugging this into (3.10), we find that

[TABLE]

Under the assumption of independence of $(X,\varepsilon)$, we can rewrite (3.3.1) as:

[TABLE]

We find the LRR by using this $Q^{\mathsf{LRR}}(\beta,\gamma)$ in (3.12).

When $(\varepsilon_{1},\varepsilon_{2})$ fall in the region $A_{4,\beta,\gamma}(x)\cap A_{3,\beta,\gamma}(x)$, the game exhibits multiple equilibria. Our counterfactuals are most robust if this region is minimal. We cannot altogether disregard this region, because observations in the data may permit this region and it is possible they may arise in the counterfactual setting. Nevertheless, we can consider the value of the nuisance parameter(s) (in their identified set) that minimize the average region of multiple equilibria for the counterfactuals. (See Figure 1 for an illustration of this observation.)

3.3.2. A Switching Regression Model with Interval Outcomes

Let us look for $Q^{\mathsf{LRR}}(\beta,\gamma)$ for the example from Section 2.3.2. We take $G$ to be a discrete distribution over $\{0,1\}$ and $\mu$ be the uniform distribution over $\{0,1\}$. Hence if we write

[TABLE]

we find that

[TABLE]

Therefore, given a counterfactual distribution $\tilde{F}(x,\varepsilon)$ of $(X,\varepsilon)$,

[TABLE]

where $\tilde{F}_{X_{1},D}$ denotes the joint distribution of $(X_{1},D)$ under $\tilde{F}$. Thus, the $\kappa$-LRR of $\gamma_{0}$ can be rewritten as

[TABLE]

where $\Gamma_{P}$ is as defined in (2.6). Then we obtain the LRR set:

[TABLE]

As mentioned previously, using this LRR set, one may construct the LRR set for other target parameters. Suppose that our ultimate interest is the Average Treatment Effect (ATE) denoted by $\psi(\beta_{0},\gamma_{0},\pi_{0})$, where

[TABLE]

where $\alpha(X_{1},D;\beta,\gamma,\pi)$ is as defined in (2.7). In this case, the LRR set for $\psi(\beta_{0},\gamma_{0},\pi_{0})$ is given by

[TABLE]

where $\Theta_{\kappa,P}^{\mathsf{LRR}}(\pi)$ is the same as $\Theta_{\kappa,P}^{\mathsf{LRR}}$, except that $\Theta_{P}$ is replaced by

[TABLE]

4. Inference with the Locally Robust Refinement

4.1. Inference from Moment Inequality Models

For the sake of concreteness, we focus on the case where the parameters are identified through moment inequality models and estimated using their sample moments.999Our LRR approach does not depend on a particular way in which partial identification of a parameter arises (i.e. whether by moment inequality models or alternative models). Hence, one can combine the LRR approach with other forms of partial identification by refining the identified set using the LRR criterion proposed in this paper. For the sake of concreteness, we focus on the case where partial identification arises from moment inequality restrictions as this is the way partial identification often arises in practice. Let us consider the following moment inequality models:

[TABLE]

for an observed i.i.d. random vector $Z_{i}$, where $m_{j}(\cdot;\beta_{0},\gamma_{0})$’s are moment functions. We assume that the moment restrictions define the identified set $\Theta_{P}$ so that for all $(\beta,\gamma)\in\Theta_{P}$, the moment restrictions are satisfied.

Define the sample analogue of the moment in (4.1):

[TABLE]

Let $\hat{\sigma}_{j}^{2}(\beta,\gamma)$ be the estimated variance of $m_{j}(Z_{i};\beta,\gamma)$, i.e.,

[TABLE]

We let for $\kappa\geq 0$ (for instance, $\kappa=0.01,0.02$ or $0.03$)101010The parameter $\kappa$ makes the inference slightly more conservative, which facilitates the development of the uniform inference (see the Supplemental Note). The choice of a positive constant $\kappa$ does not affect the uniform validity of the inference, as long as it is a fixed constant. In practice, values of $\kappa$ between 0.01 to 0.03 appear to work very well, as we show in the Monte Carlo simulations in Section 4 and the empirical application (Section 5).

[TABLE]

where $[a]_{+}=\max\{a,0\}$, $a\in\mathbf{R}$. When $\kappa=0$, we simply write $\hat{Q}(\beta,\gamma)=\hat{Q}_{0}(\beta,\gamma)$. Define

[TABLE]

Then, we construct $T(\beta,\gamma)$ as

[TABLE]

As for critical values, we may consider two different approaches. The first approach is based on least favorable configurations. The second approach is a more refined version with enhanced power properties but with a higher computational cost.

As for the first approach, we consider the following:111111We take this form of a test statistic for the sake of concreteness. One can use different functionals to produce different test statistics. See Andrews/Shi:13:Eca for various functionals.

[TABLE]

where

[TABLE]

and $Z_{i}^{*}$’s are resampled from the empirical distribution of $Z_{i}$’s with replacement. Let

[TABLE]

We take critical values $\hat{c}_{1-\alpha}(\beta,\gamma)$ to be the $1-\alpha$ quantile of the bootstrap distribution of $\hat{T}^{*}(\beta,\gamma)$. Then, we construct the confidence region for $(\beta_{0},\gamma_{0})$ as follows:

[TABLE]

where

[TABLE]

and $\hat{\Gamma}_{\kappa}(\beta)$ is as defined in (4.5), and $\hat{Q}^{\mathsf{LRR}}(\beta,\gamma)$ denotes a consistent estimator of $Q^{\mathsf{LRR}}(\beta,\gamma)$ defined in (3.10). (For example, in models of entry games or interval observations, this consistent estimator can be obtained by applying the sample analogue principle to the moments in (3.21) and (3.23), once one parametrically specifies the distributions of $\varepsilon_{1},\varepsilon_{2},$ and $\varepsilon$.) We take the infimum of an empty set to be infinity. With $\gamma$ restricted to be in $\hat{\Gamma}_{\kappa}^{\mathsf{LRR}}(\beta)$, our confidence interval reflects the focus on the parameter values obeying the LRR hypothesis.

As for the second approach, there are various alternative ways to construct inference that improves upon the one based on least favorable configurations. For example, one may use moment selection procedures as in Hansen:05:ET, Andrews/Soares:10:Eca and Andrews/Shi:13:Eca, the contact set method in Linton/Song/Whang:10:JOE, or a Bonferroni-based procedure as in Romano/Shaikh/Wolf:14:Eca. Here we adapt the Bonferroni-based procedure to our context.

First, we take $0<\alpha_{1}<\alpha$ (say, $\alpha_{1}=0.005$) and find $\hat{\kappa}_{\alpha_{1}}(\beta,\gamma)$ using a bootstrap procedure. We define $\hat{\kappa}_{\alpha_{1}}(\beta,\gamma)$ as the $\alpha_{1}$ quantile of the bootstrap distribution of

[TABLE]

which uses $\overline{m}_{j}^{*}(\beta,\gamma)$ defined in (4.8). The uniform asymptotic validity of such a bootstrap procedure is well known in the literature. (See also Section A.2 in the Supplemental Note for details.) Define

[TABLE]

Let

[TABLE]

We construct a bootstrap test statistic as follows:

[TABLE]

We take critical values $\tilde{c}_{1-\alpha+\alpha_{1}}(\beta,\gamma)$ to be the $1-\alpha+\alpha_{1}$ quantile of the bootstrap distribution of $\tilde{T}^{*}(\beta,\gamma)$. Then, we construct the confidence region for $(\beta_{0},\gamma_{0})$ as follows:

[TABLE]

In the Supplemental Note, we establish the uniform asymptotic validity of the confidence region. In the next section, we describe how to implement this procedure for Example 3.3.2 with both approaches to critical values.

As emphasized in the introduction, the LRR-based confidence region $\tilde{C}^{\mathsf{LRR}}_{1-\alpha}$ should not be confused with one based on the identified set. This LRR-based confidence region does not reflect the empirical content of the model. Rather it shows the confidence region when the true parameter belongs to the LRR set, that is, the true parameter belongs to the set of parameter values at which the ASF is robust to the local perturbations of the equilibrium selection rules. One may use $\tilde{C}^{\mathsf{LRR}}_{1-\alpha}$ as a complement to the confidence region based on the identified set for $(\beta_{0},\gamma_{0})$ in empirical and policy analysis.

4.2. Monte Carlo Simulations

4.2.1. Simulation Design

In this section, we present results from a Monte Carlo simulation study on the finite sample properties of the inference procedure developed previously. We focus on the switching regression model example considered in Section 3.3.2, which will be the main specification of our empirical application introduced in the next section.

Let $\tilde{Y}_{1},\tilde{Y}_{2}$ be as in Section 2.3.2 and $\alpha(X_{1},D;\beta_{0},\gamma_{0},\pi)$ as in (2.7) with the following specification, also motivated by our empirical application:

[TABLE]

We take $X_{1}=1$ for simplicity, we draw $\eta$ independently and with equal probability (i.e. $\pi_{0}=0.5$, implying an equal share of $\eta=1$ and $\eta=0$ in the population), and treatment is drawn independently for all $i$ with $D_{i}=1$ with probability 0.3. The moment inequalities we have are as follows121212 These inequalities can be generalized for a multidimensional $X_{1}$ by changing the indicator function on the left hand side to accommodate the different values of $X_{1}$.: for $x_{1}=1$ and $d\in\{0,1\}$,

[TABLE]

We consider two cases in our simulations: one where parameters $(\gamma_{0},\beta_{0},\pi_{0})$ are such that the moments are close to equalities, and the other where they are not. For the first case, which we call Specification 1, we set $(\gamma_{0},\beta_{0},Z_{1},Z_{2})=(1,0.15,2.4,7)$. The choice of $\beta_{0}=0.15$ implies a return to treatment (for $\eta=1$) equal to 15%. This specification corresponds to approximately 5% of observations being top-coded.131313As can be seen by the identified set (2.6), setting one moment as an equality leaves the other moment as a strict inequality. Therefore, we do not consider a “least-favorable” set-up with all moments holding as equalities. The second specification uses parameter values for $(\gamma_{0},\beta_{0},Z_{1},Z_{2})=(1,0.15,1.5,7)$ so that the moment inequalities are far from binding. This is done by keeping $(\gamma_{0},\beta_{0},Z_{2})$ the same as the first specification, but decreasing $Z_{1}$ so that there are now approximately 10% of observations top-coded, since we have an increased range of outcomes that are unobserved. In general, the identified set from the second specification will be larger than the first.

For both cases, we show results for the identified set and for using LRR with and without the Bonferroni-based method of Romano/Shaikh/Wolf:14:Eca. For the LRR, we set $\kappa=0.03$ throughout. The confidence interval follows the computation in (4.16). We set the nominal size at 0.05, the simulation number to 500, and the bootstrap number to 999 under a sample size of $n=500$. In this set-up, our parameter space is such that $\beta_{0}\geq 0$ and the ATE given in (3.27) equals 0.145.

4.2.2. Results

The results across specifications and procedures are shown in Table 1. They show the coverage of both the identified set and the LRR, as well as the length of the confidence sets. From the results, it is immediate that the LRR provides a smaller set of ATE $\psi(\beta_{0},\gamma_{0},\pi_{0})$, implying less variation on predicted (counterfactual) outcomes $Y_{i}$. (The identified set, on the other hand, presents all possible values consistent with the data.) The coverage of the theoretical LRR is appropriate (at least 95%) for values in the interior of the set, calculated using equations (3.23) and (3.25).141414In simulations available with the authors, the results improve as $n$ grows, as can be expected by our theoretical results. However, the computational cost also grows in these cases, particularly for the Bonferroni-based modification.

[INSERT Table 1 HERE]

We also find that the Bonferroni-based method of Romano/Shaikh/Wolf:14:Eca is effective at reducing the length of the confidence intervals, particularly for the identified set. It is most effective when the DGP is away from the configuration that is closest to moment equalities. However, this modification comes at the computational cost of calculating the term $\hat{\kappa}_{\alpha_{1}}$ via a bootstrap procedure for every value of $(\beta,\gamma)$ in a first step. This step could be time consuming when the dimension of $\gamma$ is large.

4.3. Predictions Using the LRR and the Identified Set

We conduct two additional Monte Carlo exercises to provide additional insights on the benefits of using the Locally Robust Refinement.

Our first exercise compares the predictions for our parameter of interest ($\psi$) when using the LRR compared to using the identified set across Specifications 1 and 2. It illustrates when the LRR’s predictions are closest to the true parameter values and when they are most informative (i.e. when the predictions vary the least).

To do so, we draw $s=1,...,1000$ values $\psi^{s}$ from the average (conservative) 95% confidence set for the LRR or the identified set described in Table 1. These sets would be the predictions reported by an empirical researcher, who does not observe or point-identify the true $\psi(\beta_{0},\gamma_{0},\pi_{0})$ due to top-coding. We draw these vectors in 2 different ways: (i) uniformly (i.e. any parameter within the appropriate set is drawn with equal probability) and (ii) furthest-case (i.e. we draw the parameters in each set that generate average predictions furthest away from the true values).151515These sampling schemes represent different scenarios that illustrate the performance of our refinement. They may be thought of as different researcher priors on the true parameter. One can also perform the exercise under intermediate weighting schemes, obtaining similar conclusions. To showcase the results, in Figure 2, we report the distribution of the absolute deviation $\left|\psi^{s}-\psi(\beta_{0},\gamma_{0},\pi_{0})\right|$: i.e. how do the predictions using the LRR and the identified set differ from the true value.

[INSERT FIGURES 2 AND 3 HERE]

As we can see, the LRR’s predictions vary less than those with the identified set (they are often at least 30% shorter), regardless of the specification or how the parameters are drawn. The use of the LRR seems particularly favorable when (i) the identified set is large (Specification 2), and (ii) when there is a significant chance of drawing extreme values of parameters from the identified set. This difference is most salient when the furthest-care scenario is particularly important to the researcher (for instance, when a bad policy could be very costly in terms of welfare). Afterall, the LRR’s objective is not about mean predictions, but rather as minimizing the worst case.

The specifications above satisfy the LRR hypothesis, since the true $\psi(\beta_{0},\gamma_{0},\pi_{0})$ is in the LRR set. Does the LRR still preserve these favorable properties even if the LRR hypothesis is not satisfied? We investigate this question in a last exercise. We set $(\beta_{0},Z_{1},Z_{2})$ to the same values as Specification 1, but we increase $\gamma_{0}$ to 2.5. As a result, $\gamma_{0}$ is no longer in the LRR. We then compare the performance of the LRR and of the identified set in predicting the Average Structural Outcome (ASO) obtained using (2.8).161616In this setting, $\psi(\beta_{0},\gamma_{0},\pi_{0})$ will generally satisfy the LRR hypothesis due to the parametrizations in this example. Hence as a target parameter that is not in the LRR set in this design, we consider the ASO in this exercise. This outcome is unobserved, but it can be predicted based on the LRR and the identified set. The results are shown in Figure 3. In contrast to Figure 2, some draws of the identified set can outpredict the LRR. Afterall, the former contains the true parameters which are now absent in the LRR. Nevertheless, the LRR still generates more stable predictions, and its gains are still most noticeable and relevant when the identified set is large and/or worst-case outcomes are likely. Hence, while our results rely on the simulation designs motivated by our empirical application, the approach of LRR seems to generates predictions with stable performance regardless of whether the LRR hypothesis is satisfied or not.

5. Empirical Illustration

We provide an empirical application to illustrate how our procedure works with data. The application considers the case of top-coded data, as in the Monte Carlo simulations. We focus on the returns to college education, a topic subject to a large body of work in economics (see Oreopoulos/Petronijevic:13:ERIC for a survey). For the purpose of illustration, we choose a simplified approach and focus on the ATE defined in (3.27).171717One could extend our empirical analysis to incorporate potential endogeneity of the policy variable by using instrumental variables, and considering a Local Average Treatment Effect (LATE) which is the returns of college education on wages for the subgroup of individuals who are induced to attend college by a change in the instrument. Or alternatively, one could focus on the Marginal Treatment Effect or a Marginal Policy Relevant Treatment Effect as emphasized by Carneiro/Heckman/Vytlacil:11:AER. Depending on the question of interest, such a parameter could allow the evaluation of policies meant to increase enrollment, such as changing taxes or subsidies for college (see Dynarski/Scott:17:Tax for an overview of different policies in place in the U.S. and their associated costs). These policies could have large fiscal and welfare consequences.

In this set-up, the outcome ($Y_{i}$) is the log of the hourly wages, the policy treatment is attending college (i.e. $D_{i}\in\{0,1\}$, where $D_{i}=1$ represents that $i$ attended college and [math] otherwise) and $X_{1,i}$ are individual covariates such as gender and/or race. Finally, the decision to attend college may depend on unobserved types: some types may have higher wages and/or benefit more from college. We model this by the random variable $\eta$, which is assumed to be binary: the returns from treatment $D_{i}=1$ may depend on whether unobserved type is high ($\eta=1$) or low ($\eta=0$) and this difference can exist throughout the distribution of $X_{1,i}$.181818For illustration purposes, we assume $\eta$ is binary and independent of $X_{1},D$. Both can be generalized, but that is beyond the scope of our exercise.

Since top-coding is prevalent in wage data (it is present in such widely used datasets as the NLSY and the Current Population Survey - CPS) and such top-coding is not random (for example, college educated individuals are more likely to be top-coded since they have higher wages on average), this set-up is nested within Example 3.3.2. Hence, the identified set is given in (2.6) and the LRR in (3.13) can be used with the expressions defined in and above (3.25).

We follow Romano/Shaikh:10:Eca and use data from the 2000 Annual Demographic Supplement of the Current Population Survey (CPS), keeping individuals aged 20 to 24, white, with a primary source of income from wages and salaries and that worked at least 2 hours per week on average. Akin to their work, our covariates $X_{1,i}$ are a constant and a binary variable for Male/Female. We differ from their sample by keeping both college graduates and non-college graduates, since that is our counterfactual of interest. This yields a sample of 5816 individuals, of which 826 (14%) are treated ($D_{i}=1$). Details on the data construction are provided in the Supplemental Note.191919Romano/Shaikh:10:Eca took the empirical distribution of the data as a “true” probability distribution to perform Monte Carlo simulations. Here, as in typical empirical research, we regard the data as a sample from an unknown population. We then explore the effects of different amounts of top-coding on the results.

Following the large literature on Mincerian equations (see Heckman/Humphries/Veramendi:18:JPE for a discussion), we use a linear specification for $g_{1}$:

[TABLE]

For the reduced form for $\eta=0$, we parametrize $g_{2}(\beta_{0}D+X_{1}^{\prime}\gamma_{0})=\log(g_{1}(\beta_{0}D+X_{1}^{\prime}\gamma_{0}))$.202020While this is an illustration and other specifications could be used, this nonlinearity allows for increasing differentials across types, while its nonseparability allows for heterogeneous returns to college based on observable characteristics (see Barrow/Rouse:AERPP:05 for one such study). We present confidence sets for both the identified set for the ATE, $\psi_{0}=\psi(\beta_{0},\gamma_{0},\pi_{0})$, and the LRR for $\psi_{0}$ using both conservative and a Bonferroni-correction based inference. We present results for this data under different top-coding amounts, imposing either 5% or 10% top-coding in the sample, under a value of top possible wages of $Z_{2}=10^{8}$.212121We also set the upper bound on $\beta_{0}$ to 100% (or 25% per year), above all estimates typically found in the literature (see Oreopoulos/Petronijevic:13:ERIC). This is a natural restriction using the literature’s results, and helps speed up computation as we do not search for a priori unreasonable values of $\beta_{0}$). The results are shown below. As standard, we report the returns to a year of college (i.e. $\psi_{0}/4$).

[INSERT TABLE 2 HERE]

Our estimates are largely consistent with the literature, including the ATE found in works as Carneiro/Heckman/Vytlacil:11:AER, despite the different sample. While the identified set is large, including ATE close to 0 and others close to 25%, the LRR is significantly shorter, between 3 and 11% (for 5% top-coding) and between 0 and 10% (for 10% top-coding). These results are also robust to different choices of $\kappa$.222222We used $\kappa=0.03$ which was shown to work well in the Monte Carlo simulations. When setting $\kappa=0.01$, the LRR confidence sets are very similar (and shorter, since a larger $\kappa$ is more conservative): in the conservative inference case, they are $[0.061,0.091]$ for 5% top-coding and $[0,0.091]$ for 10% top-coding.

To see the usefulness of the refinement in this context, consider one of the policies motivating this exercise. This could be a change in subsidies to college or expanding college loans with the purpose of a large increase in college attendance. In the presence of top-coding, the identified set cannot reject large rates of return for college (e.g. above 12% a year) - suggesting a rationale for financially costly policies. However, the LRR rejects these larger values, finding rates of return that are lower than 11% and more compatible with those observed in the literature. The LRR suggests that more costly counterfactual policies should not be undertaken. This rejection is not due to the informational content of the model, which is still captured by the identified set, but rather from the robustness of the counterfactual predictions. The larger returns to college found in the identified set seem to be driven by features of the model the researcher is less confident about.

6. Conclusion

This paper explores methods to deal with models with multiple reduced forms for counterfactual analysis. As mentioned by Eizenberg:14:ReStud, there is an inherent tradeoff within the partial identification literature: while researchers may prefer not to impose strong identifying assumptions, they must confront the issue of counterfactuals with larger identified sets when doing policy analysis.

This paper points out that not all the values of the parameter are equally desirable for counterfactual analysis despite their empirical relevance. In particular, this paper focuses on the robustness property of the parameter values when the reduced form selection rule is locally perturbed. Thus the refinement looks at the subset of counterfactual scenarios that give most reliable predictions against a range of perturbations of the reduced forms (such as through changing equilibrium selection rules).

There could be other ways to discriminate different values of the identified set depending on the purpose of the counterfactual analysis in the application. For example, one may consider local misspecification of causal relationships between certain variables. This is plausible when the researcher is not sure about the direction of causality between certain variables. Similarly as we do in this paper, one may want to find a refinement that is robust to this type of misspecification. A fruitful formulation of this approach is left for future research.