On the Misspecification of Linear Assumptions in Synthetic Control

TL;DR

This paper investigates the limitations of the linearity assumption in synthetic control methods, providing bounds on misspecification error and developing new estimators that incorporate additional data to improve causal inference accuracy.

Contribution

It introduces theoretical bounds on misspecification error and proposes new synthetic control estimators using supplementary data to reduce bias from linearity assumption violations.

Findings

New estimators outperform standard synthetic controls on synthetic data.

Bounds on misspecification error are small for minor violations.

Application to California tobacco data suggests true effects outside bounds.

Abstract

The synthetic control (SC) method is a popular approach for estimating treatment effects from observational panel data. It rests on a crucial assumption that we can write the treated unit as a linear combination of the untreated units. This linearity assumption, however, can be unlikely to hold in practice and, when violated, the resulting SC estimates are incorrect. In this paper we examine two questions: (1) How large can the misspecification error be? (2) How can we limit it? First, we provide theoretical bounds to quantify the misspecification error. The bounds are comforting: small misspecifications induce small errors. With these bounds in hand, we then develop new SC estimators that are specially designed to minimize misspecification error. The estimators are based on additional data about each unit, which is used to produce the SC weights. (For example, if the units are countries then the additional data might be demographic information about each.) We study our estimators on synthetic data; we find they produce more accurate causal estimates than standard synthetic controls. We then re-analyze the California tobacco-program data of the original SC paper, now including additional data from the US census about per-state demographics. Our estimators show that the observations in the pre-treatment period lie within the bounds of misspecification error, and that the observations post-treatment lie outside of those bounds. This is evidence that our SC methods have uncovered a true effect.

Tables

Table 1. Table 1 : Non-zero weights returned by the M-bound estimator for the synthetic California of Fig.   2 .

D.C	Hawaii	Nevada	New Mexico	Texas
0.106	0.166	0.195	0.209	0.324

Equations

$$\displaystyle w=\operatorname*{arg\,min}_{w\in\Delta^{J}}\sum^{T_{0}-1}_{t=0}\Bigl{(}y_{0t}-\sum_{j}{w}_{j}y_{jt}\Bigr{)}^{2}.$$

$$Y_{jt}={\mu }_j^{\top }{\lambda }_t+{ϵ}_{jt},$$

$$\exists w\in {\Delta }^J,{\mu }_0={\sum }_j{w}_j{\mu }_j,$$

$$\mathbb{E}\left[Y_{jt}\right]={\int }_x{\mathbb{E}}_t\left[Y\mid x\right]p_j(x)\text{leavevmode}\text{d}x.$$

$$\mathbb{E}\left[Y_{jt}\right]=\sum _{x_k}\underset{({\lambda }_t{)}_k}{\underbrace{{\mathbb{E}}_t\left[Y\mid x_k\right]}}\underset{({\mu }_j{)}_k}{\underbrace{p_j(x_k)}}={\lambda }_t^{\top }{\mu }_j.$$

$$\exists w\in {\Delta }^J,p_0={\sum }_j{w}_j{p}_j,$$

$$\mathbb{E}\left[Y_{0t}\right]\approx \sum _{j=1}^J{w}_j\mathbb{E}\left[Y_{jt}\right].$$

$$\left|\mathbb{E}\left[Y_{0t}\right]-\sum _{j=1}^J{w}_j\mathbb{E}\left[Y_{jt}\right]\right|\le \ell \cdot W_1\left(p_0,{\hat{p}}_0\right),$$

$$(w_{1},...,w_{J})\mapsto W_{1}\Bigl{(}p_{0},\sum_{j}w_{j}p_{j}\Bigr{)}.$$

$$\mathbb{E}\left[Y_{0t}\right]\in \left[{\hat{y}}_{0t}-M,{\hat{y}}_{0t}+M\right]\forall t,$$

$$p_j(x,z)=p_j(x)p_j(z),$$

$${\mathbb{E}}_t\left[Y|x,z\right]=g_t(x)+h_t(z).$$

$$\left|\mathbb{E}\left[Y_{0t}\right]-{\sum }_{j=1}^J{w}_j\mathbb{E}\left[Y_{jt}\right]\right|\le \ell \cdot W_1(p_0(x),{\hat{p}}_0(x))$$

$$\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}+{\max }_{u<T_0}\left|\mathbb{E}\left[Y_{0u}\right]-{\sum }_{j=1}^J{w}_j\mathbb{E}\left[Y_{ju}\right]\right|$$

$$+\inf_{\alpha\in\Delta^{T_{0}}}\left|\int_{z}\textstyle\Bigl{(}p_{0}(z)-\hat{p}_{0}(z)\Bigr{)}\Bigl{(}\mathbb{E}_{t}\left[Y|z\right]-\sum\limits_{u<T_{0}}\alpha_{u}\mathbb{E}_{u}\left[Y|z\right]\Bigr{)}\textnormal{d}z\right|.$$

$$\textstyle w\mapsto\max\limits_{t<T_{0}}\Bigl{|}y_{0t}-\sum\limits_{j=1}^{J}w_{j}y_{jt}\Bigr{|}+\lambda\cdot W_{1}\Bigl{(}p_{0},\sum_{j}w_{j}p_{j}\Bigr{)},$$

$$\forall (x,y)\in ({\mathbb{R}}^n{)}^2,|f(x)-f(y)|\le \ell \cdot \Vert x-y{\Vert }_1$$

$$\left|\mathbb{E}\left[Y_{0t}\right]-\sum _{j=1}^J{w}_j\mathbb{E}\left[Y_{jt}\right]\right|\le \ell \cdot W_1\left(p_0,{\hat{p}}_0\right),$$

$$\mathbb{E}\left[Y_{0t}\right]-\sum _{j=1}^J{w}_j\mathbb{E}\left[Y_{jt}\right]$$

$$={\int }_x{\mathbb{E}}_t\left[Y|x\right]\left(p_0(x)-{\hat{p}}_0(x)\right)\text{d}x$$

$$\le W_1(p_0,{\hat{p}}_0)\cdot \ell$$

$$\left|\mathbb{E}\left[Y_{0t}\right]-\sum_{j=1}^{J}w_{j}\mathbb{E}\left[Y_{jt}\right]\right|\leq\ell\cdot W_{1}(p_{0}(x),\hat{p}_{0}(x))+\max_{u<T_{0}}\left|\mathbb{E}\left[Y_{0u}\right]-\sum_{j=1}^{J}w_{j}\mathbb{E}\left[Y_{ju}\right]\right|+{\inf_{\alpha\in\Delta^{T_{0}}}\left|\int_{z}\textstyle\Bigl{(}p_{0}(z)-\hat{p}_{0}(z)\Bigr{)}\Bigl{(}\mathbb{E}_{t}\left[Y|z\right]-\sum\limits_{u<T_{0}}\alpha_{u}\mathbb{E}_{u}\left[Y|z\right]\Bigr{)}\textnormal{d}z\right|}.$$

$$\forall (x,z),\text{leavevmode}{f}_t(x,z)=g_t(x)+h_t(z).$$

$${\mathbb{E}}_t\left[Y|x\right]$$

$$={\int }_z\left(g_t(x)+h_t(z)\right)p_j(z)\text{d}z$$

$$=g_t(x)+{\int }_z{h}_t(z)p_j(z)\text{d}z.$$

$$\mathbb{E}\left[Y_{0t}\right]-\sum _{j=1}^J{w}_j\mathbb{E}\left[Y_{jt}\right]$$

$$={\int }_{(x,z)}(g_t(x)+h_t(z))\left(p_0(x)p_0(z)-{\sum }_j{w}_j{p}_j(x)p_j(z)\right)\text{d}x\text{d}z$$

$$=\underset{A}{\underbrace{{\int }_x{g}_t(x)\left(p_0(x)-{\sum }_j{w}_j{p}_j(x)\right)\text{d}x}}+\underset{B}{\underbrace{{\int }_z{h}_t(z)\left(p_0(z)-{\sum }_j{w}_j{p}_j(z)\right)\text{d}z}}.$$

$$|A|=\left|{\int }_x{\mathbb{E}}_t\left[Y|x\right]\left(p_0(x)-{\sum }_j{w}_j{p}_j(x)\right)\text{d}x\right|\le \ell \cdot W_1(p_0(x),{\hat{p}}_0(x)).$$

$$B$$

$$\displaystyle=\sum\limits_{u<T_{0}}\alpha_{u}\int_{z}\Bigl{(}\mathbb{E}_{u}\left[Y|z\right]p_{0}(z)-{\textstyle\sum_{j}}w_{j}\mathbb{E}_{u}\left[Y|z\right]p_{j}(z)\Bigr{)}\text{d}z+\int_{z}\Bigl{(}\mathbb{E}_{t}\left[Y|z\right]-{\textstyle\sum\limits_{u<T_{0}}}\alpha_{u}\mathbb{E}_{u}\left[Y|z\right]\Bigr{)}\left(p_{0}(z)-\hat{p}_{0}(z)\right)\text{d}z$$

$$\displaystyle=\sum\limits_{u<T_{0}}\alpha_{u}\Bigl{(}\mathbb{E}\left[Y_{0u}\right]-{\textstyle\sum_{j}}w_{j}\mathbb{E}\left[Y_{ju}\right]\Bigr{)}\text{d}z+\int_{z}\Bigl{(}\mathbb{E}_{t}\left[Y|z\right]-{\textstyle\sum\limits_{u<T_{0}}}\alpha_{u}\mathbb{E}_{u}\left[Y|z\right]\Bigr{)}\left(p_{0}(z)-\hat{p}_{0}(z)\right)\text{d}z.$$

Full text

1 Introduction

The synthetic control (SC) method is a popular approach for analyzing observational panel data to estimate causal effects [Aba21]. SC has been widely used in science [Pie+16] and social science [HPJ17], as well as for evaluating public policies [DAW19, Pin15, All+17].

The typical SC setup involves measurements of an outcome variable over time. One unit, called the target, received an intervention at a certain time. The other units, called donors, never received an intervention. The goal of SC is to estimate the target’s counterfactual outcomes. What would have happened had it not received the intervention?

Example: The panel data in Fig. 1 (left) contains cigarette sales across states and time. In 1988 California implemented a program that increased the tobacco tax by 25 cents. After 1988, how much would Californians have smoked had the program not been implemented? Here, California is the target; the other states are the donors.

The idea behind SC is to approximate the target’s control outcomes—the smoking rate in California without its policy—with a weighted combination of the donor’s control outcomes. In the example, SC uses data from the pre-policy periods to fit California’s pre-policy smoking rates as a weighted combination of the other states’ smoking rates. It then uses its fitted weights to estimate the smoking rate in California after 1988, had the policy not been introduced. These estimates, along with California’s post-policy rates, help assess the causal effect of the policy.

What justifies this procedure? In its original formulation, [ADH10] shows that SC is justified if the control outcomes follow a linear factor model, where a per-period factor linearly combines with a per-unit factor. Following this work, [Shi+22] shows that the linear factor model itself can be justified through assumptions about the individuals within each unit (e.g., people within each state) and invariances around the causal structure of the individual-level outcomes (e.g., whether they smoke). But whether at the aggregate or individual level, these assumptions point to the same requirement: that the target need to be expressed as a linear combination of the donors.

What if this requirement is not satisfied? What if California is not a linear combination of the other states? This paper studies the practical situation where the synthetic control is misspecified. We study how to quantify this misspecification error and how to minimize it.

In detail, we derive two bounds on the SC error, the M bound and the James Bound. Both bounds build on the causal framework of [Shi+22]. It assumes a set of invariant causes, variables that govern the individual-level outcomes in the same way across units, and where the difference between the units’ outcomes involves different distributions of those causes. For example, whether someone smokes might be caused by their age and education level, and the difference between California’s and Nevada’s smoking rates lies in their different population distributions of those demographic variables. Our theory shows how the similarity between the true target distribution of the causes and its synthetic distribution, induced by the SC weights, helps bound the error of the corresponding SC estimates.

We then consider a situation where we additionally observe external data about the invariant causes, such as demographic information about each state. We show how to use such data to estimate the misspecification interval for a fixed set of SC weights, and we develop two new algorithms for estimating SC weights that explicitly minimize the width of this interval. (One algorithm assumes we observe all invariant causes; the other does not make that assumption, but provides wider misspecification intervals.)

Thus this paper provides a new form of SC analysis, one where we analyze panel data and demographic data together to estimate the target counterfactual and assess its robustness to misspecification. Figure 2 illustrates this analysis on the California tobacco data, now also using additional data from the U.S. census about per-state demographics. Our estimators show that the observed outcome in the pre-policy period lies within the bounds of misspecification error, and that the observed outcome post-policy lies outside of those bounds. These results suggest that California’s 1988 anti-tobacco program had a true effect, despite possible misspecification of the synthetic control.

Related Work.  This paper contributes to the literature on synthetic controls [AG03, ADH10]. The M-bound and James-bound estimators of Sections 3 and 4 contribute to research on novel SC estimators [AI11, ADH15, DI16, Xu17, ASS18, Amj+19, Li20, Ath+21, IKM21]. Notably, several estimators penalize the optimization objective [AL21] or adjust the weights [Kel+21] to select donors with outcomes similar to the target. Including covariates in the estimator is recommended in [AG03] but is not mathematically justified. In contrast, the M and James bounds justify why selecting similar donors is important for estimation.

This paper also contributes to the literature on SC methods that are robust to misspecifications. Many existing works make additional assumptions; for example, outcomes follow linear factor models [BMFR21], latent factors are perfectly matched [ASS18, Pow18, FP21], treatments are assigned randomly [ADH15], or many time points are observed [CWZ21]. In contrast, we leverage external data to quantify the errors.

2 Synthetic Controls and Misspecification

We review the assumptions behind synthetic controls, the fine-grained model of [Shi+22], and formulate the problem of how to characterize the misspecification error induced by violation of the linearity assumptions.

2.1 Panel Data and a Causal Question

Consider a panel dataset containing outcome measurements $y_{jt}$ for units $j\in\llbracket 0,J\rrbracket$ over time periods $t\in\llbracket 0,T\rrbracket$. Unit $j=0$ is the target. It has received an intervention at $T_{0}$ that may have affected its outcomes $y_{0t}$ for $t\geq T_{0}$. The remaining units are donors. They did not receive an intervention.

For each unit and time, define a pair of potential outcomes $(Y_{jt},\widetilde{Y}_{jt})$, where $\widetilde{Y}_{jt}$ is the potential outcome of unit $j$ at time $t$ in the world where $j$ received intervention at $T_{0}$, while $Y_{jt}$ is the potential outcome in a world with no intervention. For $j=0$ and $t\geq T_{0}$, $y_{jt}=\widetilde{Y}_{jt}$, otherwise, $y_{jt}=Y_{jt}$.

Our causal question is, what would the target counterfactual be, had the intervention not occurred? We would like to estimate $Y_{0t}$ for $t\geq T_{0}$.

2.2 Synthetic Controls and their Assumptions

Synthetic control methods estimate the counterfactual outcomes of the target $Y_{0t},t\geq T_{0}$ with a weighted combination of the outcomes of the donors: $Y_{0t}=\sum_{j}w_{j}y_{jt}$. The SC weights are fitted from the pre-intervention outcomes,

[TABLE]

The validity of SC relies on two conditions: (1) During the pre-intervention period, the target’s outcomes can be written as a weighted combination of the control units’ outcomes. (2) The weighted combination from the pre-intervention periods generalizes to the post-intervention periods.

As a first step to obtain these two conditions, [ADH10] and most other works assume that the outcomes under no intervention are generated by a linear factor model [Bai09, ADH10, BMFR21, FP21]. We call it assumption A1.

A1. (Linear Factor Model)

Under no intervention, the outcomes are generated from a linear factor model,

[TABLE]

where $\mu_{j}$ is a unit-specific latent factor, $\lambda_{t}$ is a time-dependent factor, and $\epsilon_{jt}$ is independent random noise.

Then, [ADH10] assumes that the target’s outcomes can be written as a convex combination of the donors’ outcomes. This implies that the target’s latent factor is a convex combination of the donors’ latent factors, we it call A2.

A2. (Convex Combination)

The target unit’s latent factor is a convex combination of the donors’ latent factors,

[TABLE]

where $\Delta^{J}$ is the simplex over $J$ coordinates.

Remark 1**.**

To be precise, [ADH10] assumes that $A:=\sum_{t<T_{0}}\lambda_{t}^{\top}\lambda_{t}$ is nonsingular. Consequently, once they assume $Y_{0t}=\sum_{j}w_{j}Y_{jt}$ for $t<T_{0}$, then the invertability of $A$ implies $\mu_{0}=\sum_{j}w_{j}\mu_{j}$ (intuitively, we “invert” the factor model). The target is a convex combination of donors.

With assumptions A1 and A2 in hand, estimator (1) will identify the weights of A2. These weights will then estimate the untreated potential outcomes using the factor model A1. With assumptions A1 and A2, synthetic control is possible.

2.3 A Fine-grained Model for SC

When does the linear factor model assumption A1 hold in practice? [Shi+22] explores a justification.

First, the authors notice that SC often considers large units composed of multiple individuals (states, countries) and aggregated outcomes that are averages of individual-level outcomes (per-capita cigarette consumption). Therefore, they propose a “fine-grained" model of synthetic controls, which introduces individual-level variables. The variable $Y_{ijt}$ denotes individual $i$’s outcome in unit $j$ at time $t$ (their cigarette consumption at time $t$). The outcome of each unit $Y_{jt}$ is the average of the individual outcomes in the unit.

Second, [Shi+22] posits the idea of invariant causes, which we denote as $x_{ijt}$. The invariant causes $x_{ijt}$ are individual-level variables that follow two invariance assumptions. (1) When conditioned on the invariant causes, the individual-level outcomes do not depend on which unit the individual is from. This way, $x\mapsto\mathbb{E}\left[Y_{ijt}|x_{ijt}=x\right]$ is the same function for all $i$ and $j$; we write it $\mathbb{E}_{t}\left[Y|x\right]$. (2) The distribution of the invariant causes in each unit can change from unit to unit but remain the same across time. The distribution of invariant causes over the individuals in unit $j$ is denoted $x\mapsto p_{j}(x)$, with no dependence in $t$.

Finally, with these individual-level variables, [Shi+22] shows that the unit-level outcomes are

[TABLE]

Further, if the distributions of invariant causes $p_{j}$ are discrete and finite then Eq. 3 becomes a finite sum,

[TABLE]

Thus Equation 4 justifies the linear factor model A1 and provides context about what the latent factors might represent.

Regarding A2, [Shi+22] still assumes it; it rewrites,

[TABLE]

where the equality is in the space of probability distributions.

2.4 Misspecification in SC

[Shi+22] explains where the linear factor model A1 can come from. However, even with the fine-grained approach, the factor model arises from the strong assumption of discrete and finite causes, which might not hold in practice. In other words, the factor model A1 is not guaranteed.

The convex combination A2 is also a key assumption that is unlikely to hold in practice. With a limited number of donors, the $p_{j}$ (or $\mu_{j}$) can be linearly independent. And with continuous invariant causes, the $p_{j}$ are densities, making it impossible to match $p_{0}$ even with infinitely many donors.

In this work, we relax A1 and A2. To relax A1, we use the fine-grained model of [Shi+22] but do not assume that causes are discrete and finite. To relax A2, is the focus of the paper. When A2 is violated, the target is not a convex combination of the donors. The synthetic control is misspecified, which leads to errors in the estimation of causal effects. Formally, we define the misspecification error as the absolute difference between the expected counterfactual outcome and the synthetic outcome (after intervention):

$\left|\mathbb{E}\left[Y_{0t}\right]-\sum_{j=1}^{J}w_{j}\mathbb{E}\left[Y_{jt}\right]\right|$ for $t\geq T_{0}$.

We turn to the problem of how to bound and minimize this error. We will show how to leverage external data about the invariant causes – such as demographic data from the census – to estimate a misspecification interval for the synthetic outcomes. We then derive new ways to fit the SC weights that minimize the width of that interval. The result is a new type of estimate of the SC counterfactual, and an assessment of its sensitivity to the linearity misspecification of A2.

3 The M Bound and its Estimator

In this section, we derive an exact bound that quantifies errors induced by violation of A2. Using this bound, we develop an estimator minimizing A2 misspecification errors.

3.1 The M Bound

Let $\hat{p}_{0}$ be the synthetic distribution, defined as $\hat{p}_{0}(w)=\sum_{j}w_{j}p_{j}$. We examine the difference between the distribution of the target unit $p_{0}$ and the synthetic distribution $\hat{p}_{0}$. If $p_{0}\neq\hat{p}_{0}$ but $\hat{p}_{0}$ remains “close” to $p_{0}$, we expect the synthetic control estimate to remain approximately correct,

[TABLE]

We formalize this intuition by bounding the errors resulting from the misspecification of A2.

Bound 1** (M bound).**

For any $t$, assume that $x\mapsto\mathbb{E}_{t}\left[Y|x\right]$ is $\ell$-Lipschitz111See Appendix A for details about Lipschitz functions.. Then for any weights in the simplex $w$, we have the Misspecification error bound (M-bound):

[TABLE]

where $\hat{p}_{0}=\sum_{j}w_{j}p_{j}$ and $W_{1}$ is a $\ell_{1}$-Wasserstein distance.

The proof is in Appendix A.

The Wasserstein distance $W_{1}$ is a distance between probability distributions [Vil09]. It quantifies the differences between the true population distribution $p_{0}$ and the synthetic population distribution $\hat{p}_{0}$.

For any set of weights $w$, the M bound (1) circumscribes the error of the SC estimate by a function of the weights, the population distributions of each unit (the $p_{j}$), and the sensitivity of the outcome variables to the variation of the causes (the Lipschitz constant $\ell$).

If $p_{0}=\hat{p}_{0}$, then the Wasserstein distance W_{1}\bigl{(}p_{0},\hat{p}_{0}) between the true and the synthetic distribution is zero. The M bound recovers that the SC estimate is correct. When $p_{0}\neq\hat{p}_{0}$, the M bound shows that the estimation error is proportional to the distance W_{1}\bigl{(}p_{0},\hat{p}_{0}).

The intuition behind Eq. 5 is that when a misspecification occurs, a portion of the population $p_{0}$ is approximated with an incorrect portion of the synthetic population $\hat{p}_{0}$. It is unpredictable how these populations will behave. In the worst case, their outcomes can differ by at most the distance between them (captured by $W_{1}$) and the maximum possible variation of the conditional outcome (captured by $\ell$). Hence, the M bound proves (theoretically) that a small misspecification induces a small estimation error.

3.2 The M-bound Estimator

We established the M bound, which quantifies the misspecification error for any set of weights $w$. To find weights with minimal misspecification error, we develop the M-bound estimator. See Algorithm 1.

The M-bound estimator takes population distribution data $p_{j}$ for each unit as input and returns a set of weights that minimizes the M bound. To obtain the weights, it uses projected gradient descent with the following objective,

[TABLE]

Notice it computes the SC weights using the population distribution of each unit. It does not use the outcomes data.

After obtaining a set of weights from Algorithm 1, we can use Eq. 5 with an estimated constant $\ell$ to create a misspecification interval around the synthetic control estimate,

[TABLE]

where $\hat{y}_{0t}:=\sum_{j=1}^{J}w_{j}y_{jt},\leavevmode\nobreak\ M:=\ell\cdot W_{1}\left(p_{0},\hat{p}_{0}\right)$. Thus, the M bound, with its associated estimator and misspecification interval, can be used to discover causal effects.

In Section 5.2, we revisit the California tobacco example. We use demographic data of each US state to form the invariant causes distributions $p_{j}$ and fit the M-bound estimator with these $p_{j}$. Like standard SC, the weights returned by the estimator are used to form the synthetic outcomes. In addition, the M bound provides misspecification intervals accounting for the A2 misspecification error. Fig. 2 illustrates the synthetic control estimate with its misspecification interval generated by the M-bound estimator. We see that California’s observed outcomes lie within the interval before intervention and escape it after the intervention. This suggests that a causal effect is present, even in case of misspecification.

4 The James Bound and its Estimator

In Section 3, we derived a theoretical bound on misspecification error and showed how to use the M-bound estimator to detect a causal effect. In theory, the true outcome is guaranteed to lie within the M bound. In practice, the misspecification interval produced by the M bound is only valid if we observe the distribution of all invariant causes $p_{j}$. Observing all invariant causes is a strong assumption that may not hold.

Here, we consider the setting where the invariant causes are only partially observed. We first derive a new error bound, the James bound, that accounts for misspecification on both the observed and unobserved causes. The James bound leverages the pre-intervention outcome data to estimate the influence of the unobserved causes on the outcome variable. To find the weights that minimize the James bound, we develop the James-bound estimator. Finally, we discuss when it is appropriate to use the M bound versus the James bound.

4.1 The James Bound

So far, we have used $x$ to denote all the invariant causes. With a redefinition of notation, we now refer to the observed causes as $x$, and the unobserved causes as $z$. Such that Eq. 3 becomes $\mathbb{E}\left[Y_{jt}\right]=\int_{(x,z)}p_{j}(x,z)\mathbb{E}_{t}\left[Y|x,z\right]\text{d}x\text{d}z$.

In general, we cannot bound the effect of unobserved variables without further assumptions. Here, we assume that the unobserved causes and observed causes are independent and that their respective effect on the outcome can be decomposed into two distinct terms, this is A3. We note that A1, which we relaxed, was more restrictive than A3.

A3. Independence of Observed and Unobserved Causes.

For each unit $j$, the variable $x$ and $z$ are independent,

[TABLE]

and for each time $t$, there exist functions $g$ and $h$ such that:

[TABLE]

We note that the distributions of the observed causes $x\mapsto p_{j}(x)$ and the unobserved causes $z\mapsto p_{j}(z)$ remain arbitrary, and so are $g_{t}$ and $h_{t}$.

With A3, we have “just another misspecification error” bound, the James bound.

Bound 2** (James bound).**

For $t\geq T_{0}$, assume that $x\mapsto\mathbb{E}_{t}\left[Y|x\right]$ is $\ell$-Lipschitz. Then for any weights $w\in\Delta^{J}$,

[TABLE]

[TABLE]

[TABLE]

The proof is in Appendix A.

The first term (7) mirrors the M bound. It quantifies the similarity between the target and synthetic unit’s distributions of observed causes, $x\mapsto p_{0}(x)$ and $x\mapsto\hat{p}_{0}(x)$.

The second term (8) measures the goodness of fit of the pre-intervention outcomes. It indirectly estimates the similarity between the target and the synthetic unit’s distributions of unobserved causes, $z\mapsto p_{j}(z)$ and $z\mapsto\hat{p}_{j}(z)$.

The last term (9) contains the remaining error terms. We cannot compute this term directly because it contains unobserved quantities. In Appendix A, we argue that this term is small, and we may ignore it in practice. We also present two explicit models in which the term is null and show that it is null in the standard SC factor model.

4.2 The James-bound Estimator

Building on the James bound (2), we derive the James-bound estimator. The estimator identifies the weights that minimize the following objective,

[TABLE]

where $\lambda$ is a hyperparameter. We update Algorithm 1 into a minimization algorithm for this objective function, it is reported in Appendix A.

If hyperparameter $\lambda$ is set to $\ell$, and if term (9) is effectively negligible, then Eq. 10 is precisely the James bound. Otherwise, it can be viewed as the pre-intervention errors (first term), regularized by the Wasserstein distance over external data (second term). With this perspective, the James-bound estimator finds weights that minimize pre-intervention errors while favoring donors that are similar to the target.

4.3 Choosing between M and James Bound

We introduced two bounds, along with associated estimators and misspecification intervals. M bounds are tighter but require data about all the invariant causes. James bounds can be wider but require fewer data.

As a practical guide, we recommend using the James-bound estimator first. It is indeed more prudent to assume that some invariant causes might be unobserved. If the post-intervention target outcomes fall outside the misspecification interval, we have discovered a causal effect robust to A2 misspecification (see Fig. 2).

If the post-intervention misspecification interval is too wide to detect a causal effect, then it could be that there is no causal effect. But it could also be that there is too much misspecification to use SC or that the James bound is too loose. We cannot conclude in favor of a causal effect in the first two cases. To check if the James bound is too loose and find a tighter bound, we can use the M bound.

The M bound is guaranteed valid if all invariant causes are observed. Since the M estimator does not use outcome data, the target’s pre-intervention outcomes can be used as a validation set. If the observed pre-intervention outcomes fall outside the predicted misspecification interval, not all invariant causes were observed, and we cannot apply the M bound. Otherwise, we may use the M bound.

5 Empirical Studies

We examine the M-bound and James-bound estimators using synthetic and tobacco consumption data. With synthetic data, we demonstrate that the M-bound and James-bound estimators produce better estimates in case of misspecification, and show that their misspecification intervals contain the counterfactual outcomes correctly. Using the tobacco consumption case study, we demonstrate how to collect external data and how to choose between M-bound and James-bound estimators. We find that the post-intervention California outcomes escape the misspecification error bound, suggesting that there is an actual causal effect.

Implementation Details.  To implement the algorithms we need to manipulate probability distributions and calculate Wasserstein distances with their gradients. Our implementation expects the input $p_{j}$ to be non-parametric distributions represented by a collection of atoms and associated probabilities: $p_{j}=\sum_{x\in X}\delta_{x}\cdot p_{j}(x)$, where $X$ is the set of atoms and $\delta_{x}$ is a point mass at $x$. If $p_{j}$ is discrete, such as a histogram, then the atoms are the possible values of the causes, and $p_{j}(x)$ their associated probabilities. If $p_{j}$ is continuous, then the atoms are samples of $p_{j}$, and $p_{j}(x)$ is the normalized density at $x$. For all experiments, we compute the gradients of (w_{1},...,w_{J})\mapsto W_{1}\bigl{(}p_{0},\sum_{j}w_{j}p_{j}\bigr{)}, using the Python Optimal Transport library [Fla+21] coupled with PyTorch [Pas+19]. We use gradient descent with a learning rate $\alpha=5\cdot 10^{-6}$ and $200,000$ epochs.

5.1 Experiments with Synthetic Data

Data Description.  We generate synthetic data by defining the conditional distribution $\mathbb{E}_{t}\left[Y|x\right]=f(x,t)$ and the causes distributions $p_{j}(x)$. We create six different units (called g20, g45, g50, g60, g65, g70), and consider that a single cause $x\in\mathbb{R}$ impacts the outcome $Y$. The units can be thought of as different groups of people (e.g. cities), and the cause $x$ as the age of each individual in these groups. The six units have different distributions of age (group gX has an average age of X). The target group is g45, the panel duration is $T=50$, and the intervention time is $T_{0}=15$.

The closed form equations of $(t,x)\mapsto\mathbb{E}_{t}\left[Y|x\right]$ and $(j,x)\mapsto p_{j}(x)$ are in Appendix B while Figure 3 shows the evolution of $x\mapsto\mathbb{E}_{t}\left[Y|x\right]$ over time $t$ as well as the distributions $x\mapsto p_{j}(x)$ for each unit $j$. The expected outcome $\mathbb{E}_{t}\left[Y|x\right]$ varies over time, in different ways for each $x$.

We input the distributions $p_{j}$ to Algorithm 1 and obtain the weights that minimize the M bound. As a comparison, we calculate the weights obtained from the standard SC in Eq. 1. We report the weights and the induced synthetic outcomes in Fig. 4. Furthermore, we compute $\ell=4.0$ from $x\mapsto\mathbb{E}_{t}\left[Y\mid x\right]$ (valid for all $t$). This way, we obtain the exact value of the M-bound and we can form the misspecification interval of Eq. 6, shaded on Fig. 4.

Analysis.  As shown in Fig. 4, the standard SC places a large weight on donor g20, which is a unit whose individuals are very different from g45 but with similar pre-treatment outcomes. When time increases, the individuals in g20 and g45 evolve differently and the synthetic outcome of the standard SC weights deviates away from the true outcome. In contrast, the M-bound estimator places most of the weight mass on the donor g50, which contains individuals with similar $x$ as the target g45. By doing so, the synthetic outcomes might not exactly match the g45 outcomes, but they generalize better over time. We also verify that the true outcome is always contained in the misspecification interval (Eq. 6). We repeat the analysis with the James bound and obtain the same conclusions, reported in Appendix B.

With external data, we estimated the misspecification error and limited it using the M-bound and James-bound estimators. Without external data, standard SC was incorrect.

5.2 A Case Study on Real Data

We revisit the tobacco study from [ADH10] to illustrate how to collect external data, apply different estimators, and calculate the misspecification intervals.

**Prop 99. **  A tobacco control program was passed in California in 1988, which increased tobacco taxes by 25 cents. The tax revenue was used to fund anti-tobacco campaigns. Our goal is to estimate the causal effect of the tobacco control program on California’s tobacco consumption.

The tobacco panel dataset (Fig. 1) is from the [Cen19], which provides the per capita tobacco consumption for 50 states from 1970 to 2019. The intervention of interest is the tobacco program, Prop 99. The observed outcomes for California after 1988 are under intervention. All the other observed outcomes in the dataset are assumed to be under no intervention.

External Data Collection.  First, we identify the potential causes of smoking. According to [TMF04], smoking is heavily influenced by societal and cultural factors. While these factors are difficult to measure directly, they are often correlated with demographics. Several studies have found that cigarette consumption varies significantly by age, gender, race, and ethnicity [Sak+16, Cor+22]. As a result, we use age, sex, and ethnicity/race as proxies for the causes of smoking.

We use the American Community Survey (ACS) to formulate a distribution of causes for each unit. The ACS is a demographics survey program conducted continuously by the U.S. Census Bureau [Cen20]. It reports population demographics at different geographical scales, from city boroughs to states. We accessed the ACS data with the Census Reporter API [Cen20a]. For each state, the ACS provides the joint distribution of the variables age, race, sex. Each variable is discretized into multiple bins: age into 14 bins (e.g. 15 to 17, 20 to 24 years old), race takes 8 values (Asian, Black, Native American, Pacific Islander, White non-Hispanic, White Hispanic, Mix, and Other), and sex takes 2 values (Male, Female). The joints $x\mapsto p_{j}(x)$ over these variables are defined for each state on these $14\times 8\times 2=224$ demographics combinations (atoms).

We estimate $\ell$ using additional survey data from the Tobacco Use Supplement to the Current Population Survey. This independent study collects individual demographic information along with tobacco consumption. We form the expected tobacco consumption given each invariant cause and compute the induced $\ell$. More details about the computation of Lipschitz constant can be found in Appendix B.

The M-bound Estimator.  The M-bound estimator uses our newly formed distributions $p_{0},...,p_{J}$ to compute a set of SC weights. We report the weights in Section B.4 and the SC outcomes with the misspecification interval in Fig. 2. Among the set of 50 potential donors, five obtained non-zero weights: New Mexico, Nevada, D.C, Hawaii and Texas. As expected, the M-bound estimator selected states that are similar to California. New Mexico and Nevada are geographically close and have similar demographics. Both D.C. and California have a relatively young active population. And, California is the number one destination for Hawaiians moving to the US mainland (from US census).

In Fig. 2, the solid and dotted lines denote the observed and synthetic California outcomes. The shaded areas represent the misspecification intervals. California pre-intervention outcomes fall within the estimated M-bound interval, but synthetic California is not a perfect fit, there is misspecification. In spite of the misspecification, Fig. 2 shows that the post-intervention outcomes are outside of the bounds, suggesting a causal effect.

The James-bound Estimator.  As discussed in Section 4.3, the M-bound misspecification interval is only valid if we observe all the invariant causes. According to Fig. 2, California’s pre-intervention outcomes fall within the M-bound intervals. We find, however, that when some other states are considered as the target unit, their observed outcomes before the intervention are not always within the interval.

We perform placebo tests [ADH10] where each donor is considered to be the target and a synthetic control is constructed using the other donors. Because the donors did not receive the intervention, we expect synthetic outcomes to match observed outcomes. In Fig. 5, we illustrate the comparisons for three states, Colorado, Massachusetts, New Mexico. For comparisons on all states, see Appendix B.

Fig. 5 (left) shows the synthetic outcome estimates by the M-bound estimator. Both Colorado and Massachusetts’s pre-intervention outcomes are outside of the misspecification interval. This suggests that not all invariant causes are observed. While New Mexico’s pre-intervention outcomes lie within the misspecification interval of the synthetic New Mexico, the error bound is too large to use synthetic control.

Fig. 5 (right) shows the synthetic outcome estimates using the James-bound estimator. We observe that the pre-intervention outcomes across states now all fall within the James-bound misspecification intervals, which are also wider than the M-bound intervals. After the intervention, the observed tobacco consumption in Colorado remains in the James-bound misspecification interval, suggesting the intervention had no effect. This is expected because Colorado did not implement an anti-tobacco program like California. For Massachusetts, the James-bound interval is narrow enough to detect a decrease in tobacco consumption that is not due to misspecification. In fact, this is consistent with the policies taken by this state in 1993 to raise taxes and increase its Massachusetts Tobacco Control Program.

The placebo test provides further evidence that the tobacco control program in California indeed had a causal effect on tobacco consumption. In states without tobacco control programs, their outcomes fall within the misspecification interval, whereas California’s outcome does not.

6 Discussion

We address the problem of the misspecification of linear assumptions in synthetic controls. We relax assumptions commonly assumed in the literature (A1, A2), derive two misspecification bounds, and propose corresponding estimators. The key idea is to leverage external data to bound and minimize misspecification. Each bound comes with requirements: data must be available for the M bound (observe all causes), and we identify a modeling assumption for the James bound (A3). As an area of future research, we can explore other SC estimation procedures that might be enabled by these two requirements.

Appendix A Technical Details

A.1 Lipschitz function

A function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ is said to be $\ell$-Lipschitz if

[TABLE]

More precisely, the function $f$ can be from and to any metric spaces, with their associated distance. Here we use the $L_{1}$ norm to measure the distance between $x$ and $y$ in $\mathbb{R}^{n}$, and we use the absolute value to measure the distance between $f(x)$ and $f(y)$ in $\mathbb{R}$.

A Lipschitz function is limited in how fast it can change. In the context of synthetic control, if $\ell$ is small, it implies that a change in the causes induces a small change in the outcomes. Having a small $\ell$ suggests that misspecification of the causes will have a limited impact on the outcome.

A.2 Proof of the M Bound

Bound 1 (M bound).

For any $t$, let assume that $x\mapsto\mathbb{E}_{t}\left[Y|x\right]$ is $\ell$-Lipschitz, then for any weights in the simplex $w$, we have the Misspecification error bound (M-bound):

[TABLE]

where $\hat{p}_{0}=\sum_{j}w_{j}p_{j}$ and $W_{1}$ is a $\ell_{1}$-Wasserstein distance.

Proof.

Notice that for any unit $j\in\llbracket 0,J\rrbracket$, the expected outcome writes $\mathbb{E}\left[Y_{jt}\right]=\int_{x}\mathbb{E}_{t}\left[Y|X=x\right]p_{j}(x)\text{d}x$.

Fix some weights $w_{1:J}\in\Delta^{J}$ and then by the linearity of the integral,

[TABLE]

where the inequality comes from Kantorovich duality (Theorem 5.10, [Vil09]) about Wasserstein distances. ∎

A.3 Proof of the James Bound

Bound 2 (James bound).

For $t\geq T_{0}$, let assume that $x\mapsto\mathbb{E}_{t}\left[Y|x\right]$ is $\ell$-Lipschitz, then for any weights in the simplex $w\in\Delta^{J}$, we have Just Another Misspecification Errors bound (James bound) :

[TABLE]

Before proving the James bound, we need one lemma which follows from assumption A3.

Lemma 1**.**

Suppose the distributions of causes $(x,z)\mapsto p_{j}(x,z)$ and the expected outcomes $(x,z)\mapsto\mathbb{E}_{t}\left[Y|x,z\right]$ satisfy assumption A3, then $\mathbb{E}_{t}\left[Y|x\right]=g_{t}(x)+constant$ and $\mathbb{E}_{t}\left[Y|z\right]=h_{t}(z)+constant$ where the $constant$ terms are independent of $x$ and $z$ (they can depend on $t$).

Proof.

Fix $t$ and $j$, and define the function $f$ by $f_{t}(x,z)=\mathbb{E}_{t}\left[Y\mid x,z\right]$. We suppose A3, that is $p_{j}(x,z)=p_{j}(x)p_{j}(z)$ and that there exist two functions $g_{t}$ and $h_{t}$ such that

[TABLE]

Then

[TABLE]

We notice that $\int_{z}h_{t}(z)p_{j}(z)\text{d}z$ is a constant independent of $x$ or $z$. (It is the expectation of $z\mapsto h_{t}(z)$ with respect to $z\mapsto p_{j}(z)$.)

We obtain a similar result for $\mathbb{E}_{t}\left[Y|z\right]$, $h_{t}(z)$ and the constant $\int_{z}g_{t}(x)p_{j}(x)\text{d}x$. ∎

We can now prove the James bound.

Proof.

Fix some weights $w\in\Delta^{J}$, weights $\alpha\in\Delta^{T_{0}}$ then,

[TABLE]

We have $g_{t}(x)=\mathbb{E}_{t}\left[Y|x\right]+{constant}$, the constant cancels out in $A$ and we obtain:

[TABLE]

We have $h_{t}(z)=\mathbb{E}_{t}\left[Y|z\right]+{constant}$, the constant cancels in $B$ and we obtain:

[TABLE]

For any $\alpha\in\Delta^{T_{0}}$ we have:

[TABLE]

So,

[TABLE]

Because the previous inequality hold for any $\alpha\in\Delta^{T_{0}}$, we can “take” the inf on the right term.

We obtain,

[TABLE]

This proves the James bound. ∎

A.4 Interpretation of Eq. 9

In the James bound, we can compute the following terms:

• •

$\ell\cdot W_{1}(p_{0}(x),\hat{p}_{0}(x))$ is estimated with the external data on the subset of observed causes.

• •

\max\limits_{u<T_{0}}\Bigl{|}\mathbb{E}\left[Y_{0u}\right]-{\textstyle\sum_{j}}w_{j}\mathbb{E}\left[Y_{ju}\right]\Bigr{|}\text{d}z is estimated from the outcome data.

The last term: \inf\limits_{\alpha\in\Delta^{T_{0}}}\left|{\displaystyle\int_{z}}\Bigl{(}p_{0}(z)-\sum\limits_{j}w_{j}p_{j}(z)\Bigr{)}\Bigl{(}\mathbb{E}_{t}\left[Y|z\right]-{\textstyle\sum\limits_{u<T_{0}}}\alpha_{u}\mathbb{E}_{u}\left[Y|z\right]\Bigr{)}\text{d}z\right| cannot be estimated from observed data. Hence, we defined the James-bound estimator without this term. and showed that it was minimizing the James bound only if this last term is negligible.

We give justification as to why this last term might be negligible, at least in comparison to the two other terms. If any of the two following conditions holds, the last term is 0:

If $p_{0}=\sum_{j}w_{j}p_{j}$. 2. 2.

If there exists $\alpha\in\Delta^{T_{0}}$ such that $\mathbb{E}_{t}\left[Y|z\right]-{\textstyle\sum\limits_{u<T_{0}}}\alpha_{u}\mathbb{E}_{u}\left[Y|z\right]=constant$.

Naturally, we are not expecting the first condition to hold, but at least we can hope that $p_{0}(z)-\sum_{j}w_{j}p_{j}(z)$ is of the same order of magnitude as $p_{0}(x)-\sum_{j}w_{j}p_{j}(x)$ (and so of $W_{1}(p_{0}(x),\hat{p}_{0}(x))$, the first term). If in addition $\mathbb{E}_{t}\left[Y|z\right]-{\textstyle\sum\limits_{u<T_{0}}}\alpha_{u}\mathbb{E}_{u}\left[Y|z\right]$ is small, then the last term (which is a product of two small terms) is negligible compared to the other terms of the James bound.

With this intuition, it seems possible to finding a $\alpha$ that makes the full integral close to zero should be possible.

More concretely, we give two examples of models for which the term (9) is null.

Standard SC setting

In the standard SC setting, assumption A2 is made. The practitioner assumes that there exists a set of weights $(w_{j})$ such that $p_{0}=\sum_{j}w_{j}p_{j}$ (with the notations of factor models, each $p_{j}$ is a point mass located at the latent factor $\mu_{j}$, such that $\mu_{0}=\sum_{j}w_{j}\mu_{j}$). In particular, it implies that \Bigl{(}p_{0}(z)-\sum\limits_{j}w_{j}p_{j}(z)\Bigr{)}=0 and so $(\ref{eqn:bound2-3})=0$.

Arbitrary model with linear conditional expectation.

We now assume arbitrary distributions $p_{j}$. They can be point mass on linear factors as in standard SC or arbitrary continuous distributions. A2 does not need to hold, it may be impossible to write the target as a linear combination of the donors. However, we assume that the response functions $(z\mapsto\mathbb{E}_{t}\left[Y\mid z\right])_{t}$ are of the form $\mathbb{E}_{t}\left[Y\mid z\right]=\beta_{t}^{\top}z$, with $\beta_{t}$ being able to change arbitrarily over time. Actually, we even requires the $\beta_{t}$ to change enough such that there exists $\alpha\in\Delta^{T_{0}}$ such that they are linearly independent and there exists $\beta_{t}=\sum\limits_{u<T_{0}}\alpha_{u}\beta_{u}$. In that case again, $(\ref{eqn:bound2-3})=0$.

A.5 The James-bound Estimator

We adapt the M-bound estimator algorithm from Algorithm 1 into Algorithm 2.

Appendix B Experiment Details

B.1 Simulation Details

For the synthetic experiment, we generate the outcomes under no intervention by defining the conditional expected outcomes $f:(x,t)\mapsto\mathbb{E}_{t}\left[Y|x\right]$ and the unit specific causes distributions $x\mapsto p_{j}(x)$. In this experiment, $x$ is a single scalar variable.

The function $f$ we choose is represented in Fig. 3 (top). It enjoys a closed-form expression:

[TABLE]

.

It was generated by combining Lagrange polynomials in $x$ with time varying coefficients.

For each group gXX (g20, g45, g50, g60, g65, g70), their associated distribution of causes is given by a normal distribution centered at XX (e.g. at 20 for g20), and with scale 5 (variance 25). Each distribution is represented in Fig. 3 (bottom).

Because our implementations of the M-bound estimator and James-bound estimator use non-parametric distributions represented by a collection of atoms and associated probabilities, each $p_{j}$ is more precisely defined as

[TABLE]

where $\mu_{j}=$ XX for each group gXX, and the set of atoms $X$ is $X=\{90\cdot k/199\mid k\in\llbracket 0,199\rrbracket\}$.

B.2 Evaluation of the James-bound Estimator on Synthetic Data

Fig. 6 reports the estimates and weights produced by the M-bound, James-bound, and standard SC estimators. Both M-bound and James-bound estimators select donor units that are more similar to the target. The M-bound estimator favors donor g50, a unit with individuals most similar to the target. Using the standard SC estimator, donor g20 is preferred, as it has similar outcomes, but different individuals, before the intervention. The James-bound estimator chooses mainly donor g50 with a small selection of donor g20 as it trades off between selecting donors with similar outcomes and similar individuals.

Both the M-bound and James-bound estimators produce misspecification intervals that cover the true outcomes. As expected, the James bound estimator produces a wider misspecification interval than the M bound. The James-bound estimator also produces a better fit for the observed data than the M-bound estimator. This is expected since the M-bound estimator does not consider the pre-intervention outcomes, whereas the James-bound estimator does.

B.3 Using Survey Data to Estimate the Lipschitz Constant

To compute $\ell$ for the tobacco case study, we leverage external survey data. The (smallest) Lipschitz constant of a function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ is by definition,

[TABLE]

(We use the $L_{1}$ norm over $\mathbb{R}^{n}$).

For the M and James bounds, we need to compute the Lipschitz constant of $x\mapsto\mathbb{E}_{t}\left[Y|x\right]$, that is, of the expected tobacco consumption given the causes $x$. We use additional survey data from the Tobacco Use Supplement to the Current Population Survey (TUS-CPS). This study collects individual demographic information along with tobacco consumption. We estimate the expected tobacco consumption given the invariant causes $x$ and compute the induced $\ell$ with the formula above by computing the pairwise differences, normalized by the differences of causes $x$.

Handling categorical causes.  For both the Wasserstein distance and the Lipschitz constant, we take $L_{1}$ norms over the causes $x$. Some causes might be categorical. We represent a categorical variable $C$ which can take $k$ values $c_{1},...,c_{k}$ as a one-half-hot encoding with $k$ different binary variables $x_{1},...,x_{k}$ with values [math] and $\frac{1}{2}$, such that $C=c_{r}$ is represented by $(x_{1},...x_{k})=(\mathds{1}(i=r))_{1\leq i\leq k}$.

B.4 California M-estimator Weights

B.5 Placebo tests

In Figs. 7a, 7b and 7c, we report the full placebo study with all the states.