A Proof of Orthogonal Double Machine Learning with $Z$-Estimators

TL;DR

This paper provides an alternative proof for the asymptotic properties of orthogonal Z-estimators in two-stage estimation, simplifying the understanding of their consistency and normality under certain conditions.

Contribution

It offers a simplified, expository proof of the asymptotic normality of orthogonal Z-estimators in two-stage models, extending prior results to a variant based on empirical moment conditions.

Findings

Orthogonal Z-estimators are $\,\sqrt{n}$-consistent and asymptotically normal.

Sample splitting and $n^{1/4}$-consistency of the first stage are sufficient.

The proof simplifies understanding of the estimator's asymptotic behavior.

Abstract

We consider two stage estimation with a non-parametric first stage and a generalized method of moments second stage, in a simpler setting than (Chernozhukov et al. 2016). We give an alternative proof of the theorem given in (Chernozhukov et al. 2016) that orthogonal second stage moments, sample splitting and $n^{1/4}$-consistency of the first stage, imply $\sqrt{n}$-consistency and asymptotic normality of second stage estimates. Our proof is for a variant of their estimator, which is based on the empirical version of the moment condition (Z-estimator), rather than a minimization of a norm of the empirical vector of moments (M-estimator). This note is meant primarily for expository purposes, rather than as a new technical contribution.

Equations

$$\mathbb{E}[m(Z,{\theta }_0,h_0(X))]=0$$

$$\hat{\theta }\text{ solves }:\frac{1}{n}\sum _{t=1}^{n}m(Z_t,\hat{\theta },\hat{h}(X_t))=0$$

$$\sqrt{n}(\hat{\theta }-{\theta }_0)\to N(0,\Sigma )$$

$$\mathbb{E}\left[{\nabla }_{\gamma }m(Z,{\theta }_0,h_0(X))\cdot (\hat{h}(X)-h_0(X))\right]=0$$

$$n^{1/2}{E}_X\left[\Vert \hat{h}(X)-h_0(X){\Vert }^2\right]{\to }_{p}0$$

$$\sqrt{n}(\hat{\theta }-{\theta }_0)=\underset{A}{\underbrace{{\left[\frac{1}{n}\sum _{t=1}^n{\nabla }_{\theta }m(Z_t,\tilde{\theta },\hat{h}(X_t))\right]}^{-1}}}\underset{B}{\underbrace{\frac{1}{\sqrt{n}}\sum _{t=1}^{n}m(Z_t,{\theta }_0,\hat{h}(X_t))}}$$

$$\underset{\theta \in \Theta }{\sup }\Vert \frac{1}{n}\sum _{t=1}^n{\nabla }_{\theta }m(Z_t,\theta ,\hat{h}(X))-\mathbb{E}[{\nabla }_{\theta }m(Z,\theta ,\hat{h}(X))]\Vert {\to }_{p}0$$

$$\frac{1}{n}\sum _{t=1}^n{\nabla }_{\theta }m(Z_t,\tilde{\theta },\hat{h}(X))\to \mathbb{E}\left[{\nabla }_{\theta }m(Z,{\theta }_0,\hat{h}(X)\right]$$

$$\frac{1}{n}\sum _{t=1}^n{\nabla }_{\theta }m(Z_t,\tilde{\theta },\hat{h}(X))\to \mathbb{E}\left[{\nabla }_{\theta }m(Z,{\theta }_0,h_0(X)\right]$$

$${\left[\frac{1}{n}\sum _{t=1}^n{\nabla }_{\theta }m(Z_t,\tilde{\theta },\hat{h}(X))\right]}^{-1}\to {\left[\mathbb{E}\left[{\nabla }_{\theta }m(Z,{\theta }_0,h_0(X))\right]\right]}^{-1}=J^{-1}$$

$$\begin{gathered} B=\underset{C}{\underbrace{\frac{1}{\sqrt{n}}\sum _{t=1}^{n}m(Z_t,{\theta }_0,h_0(X_t))}}+\underset{D}{\underbrace{\frac{1}{\sqrt{n}}\sum _{t=1}^n{\nabla }_{\gamma }m(Z_t,{\theta }_0,h_0(X_t))\cdot \left(\hat{h}(X_t)-h_0(X_t)\right)}} \\ +\underset{E}{\underbrace{\frac{1}{2\sqrt{n}}\sum _{t=1}^n{\left(\hat{h}(X_t)-h_0(X_t)\right)}^T{\nabla }_{\gamma \gamma }m(Z_t,{\theta }_0,\tilde{h}(X_t))\cdot \left(\hat{h}(X_t)-h_0(X_t)\right)}} \end{gathered}$$

$$|E|\le \frac{{\lambda }^{\ast }}{2}\sqrt{n}\left(\frac{1}{n}\sum _{t=1}^n\Vert \hat{h}(X_t)-h_0(X_t){\Vert }^2\right)$$

$$E[D|\hat{h}]=\sqrt{n}E\left[{\nabla }_{\gamma }m(Z,{\theta }_0,h_0(X))\cdot \left(\hat{h}(X)-h_0(X)\right)|\hat{h}\right]=0$$

$$\begin{gathered} E[D^2|\hat{h}]=\frac{1}{n}\sum _{t\ne t^{\prime }}E{\left[{\nabla }_{\gamma }m(Z,{\theta }_0,h_0(X_t))\cdot \left(\hat{h}(X)-h_0(X)\right)|\hat{h}\right]}^2 \\ +\frac{1}{n}\sum _{t=t^{\prime }}E\left[\Vert {\nabla }_{\gamma }m(Z,{\theta }_0,h_0(X))\cdot \left(\hat{h}(X)-h_0(X)\right){\Vert }^2|\hat{h}\right] \end{gathered}$$

$$E[D^2|\hat{h}]=E\left[\Vert {\nabla }_{\gamma }m(Z,{\theta }_0,h_0(X)){)}^2\cdot \left(\hat{h}(X)-h_0(X)\right){\Vert }^2\right]\le {\sigma }^{2}E\left[\Vert \hat{h}(X)-h_0(X){\Vert }^2\right]$$

$$\mathbb{E}\left[{\nabla }_{\gamma }m(Z,{\theta }_0,h_0(X))|X\right]=0$$

$$\mathbb{E}\left[{\nabla }_{\gamma }m(Z,{\theta }_0,h_0(X))\cdot (\hat{h}(X)-h_0(X))\right]=$$

$$=$$

Full text

1 Two-Stage Estimation

Suppose we have a model which predicts the following set of moment conditions:

[TABLE]

where $\theta_{0}\in R^{d}$ is a finite dimensional parameter of interest, $h_{0}:S\rightarrow R^{\ell}$ is a nuisance function we do not know, $Z$ are the observed data which are drawn from some distribution and $X\in S$ is a subvector of the observed data.

We want to understand the asymptotic properties of the following two-stage estimation process:

First stage. Estimate $h_{0}(\cdot)$ from an auxiliary data set (e.g. running some non-parametric regresssion) yielding an estimate $\hat{h}$. 2. 2.

Second stage. Use the first stage estimate $\hat{h}$ and compute an estimate $\hat{\theta}$ of $\theta_{0}$ from an empirical version of the moment condition: i.e.

[TABLE]

The question we want to ask is: is $\hat{\theta}$ $\sqrt{n}$-consistent. More formally, is it true that:

[TABLE]

for some constant co-variance matrix $\Sigma$. We will assume that the moment conditions that we use satisfy the following orthogonality property:

Definition 1** (Orthogonality).**

For any fixed estimate $\hat{h}$ that can be the outcome of the first stage estimation, the moment conditions are orthogonal if:

[TABLE]

where $\nabla_{\gamma}m(\cdot,\cdot,\cdot)$ denotes the gradient of $m$ with respect to its third argument.

2 Orthogonality Implies Root-$n$ Consistency

Assumption 1**.**

We will make the following regularity assumptions:

• •

Rate of First Stage.* The first stage estimation is $n^{-1/4}$-consistent in the squared mean-square-error sense, i.e.*

[TABLE]

where the convergence in probability statement is with respect to the auxiliary data set

• •

Regularity of First Stage.* The first stage estimate and the nuisance function are uniformly bounded by a constant, i.e.: $\|\hat{h}(x)\|,\|h_{0}(x)\|\leq C$ for all $x\in S$.*

• •

Regularity of Moments.* The following smoothness conditions hold for the moments*

For any $z,x,\gamma$ the function $m(z,\theta,\gamma)$ is continuous in $\theta$. Also $m(z,\theta,\gamma)\leq d(z)$ and $\mathbb{E}[d(Z)]<\infty$. 2. 2.

Similarly, the same conditions hold for $\nabla_{\theta}m(z,\theta,\gamma)$. 3. 3.

$\mathbb{E}\left[\nabla_{\theta}m(z,\theta_{0},h_{0}(x))\right]$* is non-singular.* 4. 4.

the Hessian $\nabla_{\gamma\gamma}m(z,\theta,\gamma)$ has the largest eigenvalue bounded by some constant $\lambda$ uniformly for all $\theta$ and $\gamma$. 5. 5.

the derivative $\nabla_{\gamma}m(z,\theta,\gamma)$ has norm, uniformly bounded by $\sigma$

Theorem 2**.**

Under Assumption 1 and assuming that $\hat{\theta}$ is consistent, if the moment conditions satisfy the orthogonality property then $\hat{\theta}$ is also $\sqrt{n}$-consistent and asymptotically normal.

Proof.

By doing a first-order Taylor expansion of the empirical moment condition around $\theta_{0}$ and by the mean value theorem, we have:

[TABLE]

where $\tilde{\theta}$ is convex combination of $\theta_{0}$ and $\hat{\theta}$. We will show that $A$ converges in probability to a constant $J^{-1}$ and that $B$ converges in distribution to a normal $N(0,V)$, for some constant co-variance matrix $V$. Then the theorem follows by invoking Slutzky’s theorem, which shows convergence in distribution to $N(0,J^{-1}V)$.

Convergence of $A$ to inverse derivative.

By the regularity of the moments, we have a uniform law of large numbers for the quantity $\frac{1}{n}\sum_{t=1}^{n}\nabla_{\theta}m(Z_{t},\theta,\hat{h}(X))$, i.e.:

[TABLE]

Since $\hat{\theta}$ is consistent, we also have that $\tilde{\theta}$ is consistent, i.e. $\tilde{\theta}\rightarrow_{p}\theta$. Combining the latter two properties, we get that conditional on the auxiliary data set:

[TABLE]

Moreover, since $\hat{h}$ is consistent we get that:

[TABLE]

Since the matrix $\mathbb{E}\left[\nabla_{\theta}m(z,\theta_{0},h_{0}(x))\right]$ is non-singular, by continuity of the inverse we get:

[TABLE]

Asymptotic normality of $B$.

To argue asymptotic normality of $B$ we take a second-order Taylor expansion of $B$ around $h_{0}(X_{t})$ for each $X_{t}$:

[TABLE]

First we observe that $C$ is the sum of $n$ i.i.d. random variables, divided by $\sqrt{n}$. Thus by the Central Limit Theorem, we get that $C\rightarrow N(0,V)$, for some constant co-variance matrix $V$. Then we conclude by showing that $D,E\rightarrow_{p}0$.

Second we argue that $n^{1/4}$ consistency of the first stage, implies that $E\rightarrow_{p}0$. Since $\nabla_{\gamma\gamma}m(z,\theta,\gamma)$ has a largest eigenvalue uniformly bounded by $\lambda^{*}$, we have that the quantity $E$ is bounded by

[TABLE]

Fixing the auxiliary data set, the quantity $\frac{1}{n}\sum_{t=1}^{n}\|\hat{h}(X_{t})-h_{0}(X_{t})\|^{2}$ converges to $\mathbb{E}[\|\hat{h}(X_{t})-h_{0}(X_{t})\|^{2}]$. Subsequently by $n^{1/4}$-consistency of the first stage, and regularity of the first stage, we get that $E\rightarrow_{p}0$.

Finally, we argue that orthogonality implies that $D\rightarrow_{p}0$. We show that both the mean and the trace of the co-variance of $D$ converge to [math]. The mean conditional on the auxiliary data set is:

[TABLE]

The diagonal entries of the co-variance conditional on the auxiliary dataset is:

[TABLE]

All the cross terms are zero by orthogonality, giving:

[TABLE]

Since $\hat{h}$ is consistent, we get that the latter converges to zero. Since the mean of $D$ and the trace of its co-variance converge to zero, we get that $D\rightarrow_{p}0$.

Consistency of the estimator also follows easily from standard arguments, if one makes Assumption 1 and the extra condition that the moment condition in the limit is satisfied only for the true parameters, which is needed for identification (see e.g. [NM94] for the formal set of extra regularity assumptions needed for consistency).

3 Orthogonal Moments for Conditional Moment Problems

One special case of when the orthogonality condition is satisfied is the following stronger, but easier to check property of conditional orthogonality:

Definition 2** (Conditional Orthogonality).**

The moment conditions are conditionally orthogonal if:

[TABLE]

Lemma 3**.**

Conditional orthogonality implies orthogonality, when an auxiliary data set is used to estimate $\hat{h}$.

Proof.

By the law of iterated expectations we have:

[TABLE]

Where in the last part we used the conditional orthogonality property.

For conditional moment problems studied in [Cha92], [CCD*+*16] shows how one can transform in an algorithmic manner an initial set of moments to a vector of orthogonal moments.