Specification testing in semi-parametric transformation models

TL;DR

This paper develops a goodness-of-fit test for semi-parametric transformation models, using a distance measure between nonparametric and parametric transformations, with asymptotic theory and bootstrap implementation.

Contribution

It introduces a novel test for assessing the fit of parametric transformation classes in semi-parametric models, including asymptotic analysis and a bootstrap procedure.

Findings

Asymptotic validity of the test under the null hypothesis.

The test can distinguish between large and small deviations from the true transformation.

A bootstrap algorithm effectively implements the proposed test.

Abstract

In transformation regression models the response is transformed before fitting a regression model to covariates and transformed response. We assume such a model where the errors are independent from the covariates and the regression function is modeled nonparametrically. We suggest a test for goodness-of-fit of a parametric transformation class based on a distance between a nonparametric transformation estimator and the parametric class. We present asymptotic theory under the null hypothesis of validity of the semi-parametric model and under local alternatives. A bootstrap algorithm is suggested in order to apply the test. We also consider relevant hypotheses to distinguish between large and small distances of the parametric transformation class to the `true' transformation.

Tables

Table 1. Table 1: Rejection probabilities at θ 0 ∈ { 0 , 0.5 , 1 , 2 } subscript 𝜃 0 0 0.5 1 2 \theta_{0}\in\{0,0.5,1,2\} and r ∈ { r 1 , r 2 , r 3 } 𝑟 subscript 𝑟 1 subscript 𝑟 2 subscript 𝑟 3 r\in\{r_{1},r_{2},r_{3}\} .

		${\theta }_0=0$		${\theta }_0=0.5$		${\theta }_0=1$		${\theta }_0=2$
level $\alpha$		$0.05$	$0.10$	$0.05$	$0.10$	$0.05$	$0.10$	$0.05$	$0.10$
$r_1$	null hyp.	0.01000	0.04000	0.03125	0.08750	0.03125	0.07750	0.01625	0.05625
	c=0.2	0.000	0.010	0.075	0.105	0.010	0.015	0.000	0.020
	c=0.4	0.000	0.000	0.020	0.045	0.000	0.015	0.120	0.200
	c=0.6	0.100	0.155	0.035	0.050	0.085	0.150	0.415	0.545
	c=0.8	0.685	0.765	0.110	0.210	0.505	0.645	0.785	0.890
	c=1	0.965	0.990	0.925	0.975	0.975	0.985	0.985	0.990
$r_2$	c=0.2	0.010	0.035	0.030	0.045	0.515	0.640	0.885	0.965
	c=0.4	0.015	0.040	0.000	0.005	0.060	0.135	0.870	0.980
	c=0.6	0.035	0.085	0.000	0.005	0.005	0.005	0.625	0.815
	c=0.8	0.020	0.040	0.010	0.040	0.000	0.005	0.185	0.325
	c=1	0.020	0.065	0.030	0.090	0.025	0.095	0.050	0.105
$r_3$	c=0.2	0.330	0.505	0.730	0.855	0.810	0.905	0.930	0.995
	c=0.4	0.730	0.865	0.815	0.945	0.875	0.970	0.915	0.990
	c=0.6	0.880	0.940	0.895	0.960	0.950	0.995	0.940	0.990
	c=0.8	0.895	0.965	0.925	0.975	0.935	0.990	0.915	0.980
	c=1	0.980	0.990	0.960	0.990	0.939	0.990	0.940	0.985

Table 2. Table 2: Rejection probabilities at θ 0 = 1 subscript 𝜃 0 1 \theta_{0}=1 and θ 0 = 2 subscript 𝜃 0 2 \theta_{0}=2 for r = r 1 𝑟 subscript 𝑟 1 r=r_{1} .

	Alternative	original framework		modified weighting
Param.	Level	$\alpha =0.05$	$\alpha =0.10$	$\alpha =0.05$	$\alpha =0.10$
${\theta }_0=1$	null hyp.	0.03125	0.07750	0.02875	0.07875
	c=0.2	0.010	0.015	0.040	0.100
	c=0.4	0.000	0.015	0.205	0.320
	c=0.6	0.085	0.150	0.590	0.715
	c=0.8	0.505	0.645	0.950	0.980
	c=1	0.975	0.985	1.000	1.000
${\theta }_0=2$	null hyp.	0.01625	0.05625	0.05500	0.10375
	c=0.2	0.000	0.020	0.225	0.350
	c=0.4	0.120	0.200	0.575	0.710
	c=0.6	0.415	0.545	0.910	0.965
	c=0.8	0.785	0.890	0.990	1.000
	c=1	0.985	0.990	0.995	1.000

Equations

$$h(Y)=g(X)+\epsilon ,$$

$$h(0)=0\text{and}h(1)=1.$$

$${\Lambda }_{{\theta }_0}(Y)=\tilde{g}(X)+\tilde{\epsilon },$$

$$h_0(\cdot )=\frac{{\Lambda }_{{\theta }_0}(\cdot )-{\Lambda }_{{\theta }_0}(0)}{{\Lambda }_{{\theta }_0}(1)-{\Lambda }_{{\theta }_0}(0)}.$$

$$H_{0}:\ h\in\bigg{\{}\frac{\Lambda_{\theta}(\cdot)-\Lambda_{\theta}(0)}{\Lambda_{\theta}(1)-\Lambda_{\theta}(0)}:\theta\in\Theta\bigg{\}}$$

$$H_{0}:\ h\in\bigg{\{}\frac{\Lambda_{\theta}(\cdot)-c_{2}}{c_{1}}:\theta\in\Theta,c_{1}\in\mathbb{R}^{+},c_{2}\in\mathbb{R}\bigg{\}}.$$

$$d({\Lambda }_{\theta },h)$$

$$d_n({\Lambda }_{\theta },\hat{h}):=\underset{c_1\in C_1,c_2\in C_2}{\min }\frac{1}{n}\sum _{j=1}^{n}w(Y_j)\{\hat{h}(Y_j)c_1+c_2-{\Lambda }_{\theta }(Y_j){\}}^2,$$

$$T_n=\underset{\theta \in \Theta }{\min }{d}_n({\Lambda }_{\theta },\hat{h})$$

$$\hat{\theta }:=\arg \underset{\theta \in \Theta }{\min }{d}_n({\Lambda }_{\theta },\hat{h})$$

$$\hat{h}(y)-h(y)=\frac{1}{n}\sum _{i=1}^n\psi (Z_i,\mathcal{T}(y))+o_P(n^{-1/2})\text{mbox}uniformlyiny\in {\mathcal{Y}}_w.$$

$$Z_i$$

$$H_{1,n}:h(\cdot )c_1+c_2={\Lambda }_{{\theta }_0}(\cdot )+n^{-1/2}r(\cdot )\text{mbox}forsome{\theta }_0\in \Theta ,c_1\in {\mathbb{R}}^{+},c_2\in \mathbb{R}\text{mbox}andsomefunctionr.$$

$$h(\cdot )$$

$$r_0(\cdot )$$

$$S_i=h(Y_i)=g(X_i)+{\epsilon }_i.$$

$$T_n=\underset{\gamma =(c_1,c_2,\theta )\in {\rm Y}}{\min }\sum _{j=1}^{n}w(Y_j)\{\hat{h}(Y_j)c_1+c_2-{\Lambda }_{\theta }(Y_j){\}}^2.$$

$${\dot{\Lambda }}_{\theta }(y)$$

$$R(s)$$

$${\Gamma }_0$$

$$\phi (z)$$

$$\zeta (z_1,z_2)$$

$$\bar{r}(s)$$

$$\tilde{\zeta }(z_1)$$

$$K\rho (z_1):=\int \rho (z_2)\zeta (z_1,z_2)d{F}_Z(z_2)$$

$$(\Lambda_{\theta_{0}}(1)-\Lambda_{\theta_{0}}(0))^{2}\Bigg{(}\sum_{k=1}^{\infty}\lambda_{k}W_{k}^{2}+W_{0}+E\left[w(h_{0}^{-1}(S_{1}))\bar{r}(S_{1})^{2}\right]\Bigg{)}.$$

$$T=({\Lambda }_{{\theta }_0}(1)-{\Lambda }_{{\theta }_0}(0){)}^2\sum _{k=1}^{\infty }{\lambda }_k{W}_k^2.$$

$$I(z):=w(h_0^{-1}(S_1){)}^{1/2}\left(\psi (z,U_1)-\phi (z{)}^t{\Gamma }_0^{-1}R(S_1)\right).$$

$$H_1:d(h,{\Lambda }_{\theta })>0\text{for all }\theta \in \Theta .$$

$$H_0^{\prime }:\underset{\theta \in \Theta }{\min }d(h,{\Lambda }_{\theta })\ge \eta \text{and}{H}_1^{\prime }:\underset{\theta \in \Theta }{\min }d(h,{\Lambda }_{\theta })<\eta .$$

$$M(\gamma )=M(c_1,c_2,\theta )=E\{w(Y)(h(Y)c_1+c_2-{\Lambda }_{\theta }(Y){)}^2\},$$

$${\gamma }_0=(c_{1,0},c_{2,0},{\theta }_0):=\arg \underset{(c_1,c_2,\theta )\in {\rm Y}}{\min }M(c_1,c_2,\theta ).$$

$$\Gamma^{\prime}=E\left[w(Y_{1})\left(\begin{array}[]{ccc}h(Y_{1})^{2}&h(Y_{1})&-h(Y_{1})\dot{\Lambda}_{\theta_{0}}(Y_{1})\\ h(Y_{1})&1&-\dot{\Lambda}_{\theta_{0}}(Y_{1})\\ -h(Y_{1})\dot{\Lambda}_{\theta_{0}}(Y_{1})^{t}&-\dot{\Lambda}_{\theta_{0}}(Y_{1})^{t}&\Gamma^{\prime}_{3,3}\end{array}\right)\right]$$

Full text

Key words: Bootstrap, goodness-of-fit test, nonparametric regression, nonparametric transformation estimator, parametric transformation class, testing relevant hypotheses, U-statistics

1 Introduction

It is very common in applications to transform data before investigation of functional dependence of variables by regression models. The aim of the transformation is to obtain a simpler model, e.g. with a specific structure of the regression function, or a homoscedastic instead of a heteroscedastic model. Typically flexible parametric classes of transformations are considered from which a suitable one is selected data-dependently. A classical example is the class of Box-Cox power transformations (see Box and Cox (1964)). For purely parametric transformation models see Carroll and Ruppert (1988) and references therein. Powell (1991) and Mu and He (2007) consider transformation quantile regression models. Nonparametric estimation of the transformation in the context of parametric regression models has been considered by Horowitz (1996) and Chen (2002), among others. Horowitz (2009) reviews estimation in transformation models with parametric regression in the cases where either the transformation or the error distribution or both are modeled nonparametrically. Linton et al. (2008) suggest a profile likelihood estimator for a parametric class of transformations, while the error distribution is estimated nonparametrically and the regression function semi-parametrically. Heuchenne et al. (2015) suggest an estimator for the error distribution in the same model. Neumeyer et al. (2016) consider profile likelihood estimation in heteroscedastic semi-parametric transformation regression models, i.e. the mean and variance function are modeled nonparametrically, while the transformation function is chosen from a parametric class. A completely nonparametric (homoscedastic) model is considered by Chiappori et al. (2015). Their approach was modified and corrected by Colling and Van Keilegom (2019). The version of the nonparametric transformation estimator considered in the latter paper was then applied by Colling and Van Keilegom (2018) to suggest a new estimator of the transformation parameter if it is assumed that the transformation belongs to a parametric class.

In general asymptotic theory for nonparametric transformation estimators is sophisticated and parametric transformation estimators show much better performance if the parametric model is true. A parametric transformation will thus lead to better estimates of the regression function. Moreover, parametric transformations are easier to interpret and allow for subsequent inference in the transformation model. For the latter purpose note that for transformation models with parametric transformation lack-of-fit tests for the regression function as well as tests for significance for covariate components have been suggested by Colling and Van Keilegom (2016), Colling and Van Keilegom (2017), Allison et al. (2018) and Kloodt and Neumeyer (2019). Those tests cannot straightforwardly be generalized to nonparametric transformation models because known estimators in that model do not allow for uniform rates of convergence over the whole real line, see Chiappori et al. (2015) and Colling and Van Keilegom (2019).

However, before applying a transformation model with parametric transformation it would be appropriate to test the goodness-of-fit of the parametric transformation class. In the context of parametric quantile regression, Mu and He (2007) suggest such a goodness-of-fit test. In the context of nonparametric mean regression Neumeyer et al. (2016) develop a goodness-of-fit test for the parametric transformation class based on an empirical independence process of pairs of residuals and covariates. The latter approach was modified by Hušková et al. (2018), who applied empirical characteristic functions. In a linear regression model with transformation of the response Szydłowski (2017) suggests a goodness-of-fit test for the parametric transformation class that is based on a distance between the nonparametric transformation estimator considered by Chen (2002) and the parametric class. We will follow a similar approach but consider a nonparametric regression model. The aim of the transformations we consider is to induce independence between errors and covariates. The null hypothesis is that the unknown transformation belongs to a parametric class. Note that when applied to the special case of a class of transformations that contains as only element the identity, our test provides indication on whether a classical homoscedastic regression model (without transformation) is appropriate or whether first the response should be transformed. Our test statistic is based on a minimum distance between a nonparametric transformation and the parametric transformations. We present the asymptotic distribution of the test statistic under the null hypothesis of a parametric transformation and under local alternatives of $n^{-1/2}$-rate. Under the null hypothesis the limit distribution is that of a degenerate U-statistic. With a flexible parametric class applying an appropriate transformation can reduce the dependence enormously, even if the ‘true’ transformation does not belong to the class. Thus, for the first time in the context of transformation goodness-of-fit tests we consider testing for so-called precise or relevant hypotheses. Here the null hypothesis is that the distance between the true transformation and the parametric class is large. If this hypothesis is rejected, then the model with the parametric transformation fits well enough to be considered for further inference. Under the new null hypothesis the test statistic is asymptotically normally distributed. The term “precise hypotheses” refers to Berger and Delampady (1987). Dette et al. (2018) considered precise hypotheses in the context of comparing mean functions in the context of functional time series. Note that the idea of precise hypotheses is related to that of equivalence tests, which originate from the field of pharmacokinetics (see Lakens (2017)). Throughout we assume that the nonparametric transformation estimator fulfills an asymptotic linear expansion. It is then shown that the estimator considered by Colling and Van Keilegom (2019) fulfills this expansion and thus can be used for evaluating the test statistic.

The remainder of the paper is organized as follows. In Section 2 we present the model and the test statistic. Asymptotic distributions under the null hypothesis of a parametric transformation class and under local alternatives are presented in Section 3, which also contains a consistency result and asymptotic results under relevant hypotheses. Section 4 presents a bootstrap algorithm and a simulation study. Appendix A contains assumptions, while Appendix B treats a specific nonparametric transformation estimator and shows that it fulfills the required conditions. The proofs of the main results are given in Appendix C. A supplement contains a rigorous treatment of bootstrap asymptotics.

2 The model and test statistic

Assume we have observed $(X_{i},Y_{i})$, $i=1,\dots,n$, which are independent with the same distribution as $(X,Y)$ that fulfill the transformation regression model

[TABLE]

where $E[\varepsilon]=0$ holds and $\varepsilon$ is independent of the covariate $X$, which is $\mathbb{R}^{d_{X}}$-valued, while $Y$ is univariate. The regression function $g$ will be modelled nonparametrically. The transformation $h:\mathbb{R}\to\mathbb{R}$ is strictly increasing. Throughout we assume that, given the joint distribution of $(X,Y)$ and some identification conditions, there exists a unique transformation $h$ such that this model is fulfilled. It then follows that the other model components are identified via $g(x)=E[h(Y)|X=x]$ and $\varepsilon=h(Y)-g(X)$. See Chiappori et al. (2015) for conditions under which the identifiability of $h$ holds. In particular conditions are required to fix location and scale and we will assume throughout that

[TABLE]

Now let $\{\Lambda_{\theta}:\theta\in\Theta\}$ be a class of strictly increasing parametric transformation functions $\Lambda_{\theta}:\mathbb{R}\to\mathbb{R}$, where $\Theta\subseteq\mathbb{R}^{d_{\Theta}}$ is a finite dimensional parameter space. Our purpose is to test whether a semi-parametric transformation model holds, i.e.

[TABLE]

for some parameter $\theta_{0}\in\Theta$, where $\tilde{\varepsilon}$ and $X$ are independent. Due to the assumed uniqueness of the transformation $h$ one obtains $h=h_{0}$ under validity of the semi-parametric model, where

[TABLE]

Thus we can write the null hypothesis as

[TABLE]

which thanks to (2.2) can be formulated equivalently as

[TABLE]

Our test statistics will be based on the following $L^{2}$-distance

[TABLE]

where $w$ is a positive weight function with compact support $\mathcal{Y}_{w}$. Its empirical counterpart is

[TABLE]

where $\hat{h}$ denotes a nonparametric estimator of the true transformation $h$ as discussed below, and $C_{1}\subset\mathbb{R}^{+}$, $C_{2}\subset\mathbb{R}$ are compact sets. Assumption (A6) in Appendix A assures that the sets are large enough to contain the true values. The test statistic is defined as

[TABLE]

and the null hypothesis should be rejected for large values of the test statistic. We will derive the asymptotic distribution under the null hypothesis and local and fixed alternatives in Section 3 and suggest a bootstrap version of the tests in Section 4.

Remark 2.1**.**

Colling and Van Keilegom (2019)** consider the estimator

[TABLE]

for the parametric transformation (assuming $H_{0}$) and observe that $\hat{\theta}$ outperforms the version without minimization over $c_{1},c_{2}$, i.e. $\tilde{\theta}=\arg\min_{\theta\in\Theta}n^{-1}\sum_{j=1}^{n}w(Y_{j})[\hat{h}(Y_{j})\{\Lambda_{\theta}(1)-\Lambda_{\theta}(0)\}+\Lambda_{\theta}(0)-\Lambda_{\theta}(Y_{j})]^{2}$ in simulations.

Nonparametric estimation of the transformation $h$ has been considered by Chiappori et al. (2015) and Colling and Van Keilegom (2019). For our main asymptotic results we need that $\hat{h}$ has a linear expansion, not only under the null hypothesis, but also under fixed alternatives and the local alternatives as defined in the next section. The linear expansion should have the form

[TABLE]

Here, $\psi$ needs to fulfil condition (A8) in Appendix A and we use the definitions ($i=1,\dots,n$)

[TABLE]

where $F_{Y}$ denotes the distribution of $Y$ and is assumed to be strictly increasing on the support of $Y$. To ensure that $\mathcal{T}$ is well defined the values [math] and $1$ are w.l.o.g. assumed to belong to the support of $Y$, but can be replaced by arbitrary values $a<b\in\mathbb{R}$ (in the support of $Y$). The expansion (2.7) could also be formulated with a linear term $n^{-1/2}\sum_{i=1}^{n}\tilde{\psi}(X_{i},Y_{i},y)$. In Appendix B we reproduce the definition of the estimator $\hat{h}$ that was suggested by Colling and Van Keilegom (2019) as modification of the estimator by Chiappori et al. (2015). We give regularity assumptions under which the desired expansion holds, see Lemma B.2. Other nonparametric estimators for the transformation that fulfill the expansion could be applied as well.

3 Asymptotic results

In this section we will derive the asymptotic distribution under the null hypothesis and under local and fixed alternatives. For the formulation of the local alternatives consider the null hypothesis as given in (2.4), i.e. $h(\cdot)c_{1}+c_{2}=\Lambda_{\theta_{0}}(\cdot)$ for some $\theta_{0}\in\Theta$, $c_{1}\in\mathbb{R}^{+}$, $c_{2}\in\mathbb{R}$, and instead assume

[TABLE]

Due to the identifiability conditions (2.2) one obtains $c_{2}=\Lambda_{\theta_{0}}(0)+n^{-1/2}r(0)$ and $c_{1}=\Lambda_{\theta_{0}}(1)-\Lambda_{\theta_{0}}(0)+n^{-1/2}(r(1)-r(0))$. Assumption (A5) yields boundedness of $r$, so that we rewrite the local alternative as

[TABLE]

where $h_{0}(\cdot)=(\Lambda_{\theta_{0}}(\cdot)-\Lambda_{\theta_{0}}(0))/(\Lambda_{\theta_{0}}(1)-\Lambda_{\theta_{0}}(0))$ and

[TABLE]

Note that the null hypothesis $H_{0}$ is included in the local alternative $H_{1,n}$ by considering $r\equiv 0$ which gives $h=h_{0}$. We assume the following data generating model under the local alternative $H_{1,n}$. Let the regression function $g$, the errors $\varepsilon_{i}$ and the covariates $X_{i}$ be independent of $n$ and define $Y_{i}=h^{-1}(g(X_{i})+\varepsilon_{i})$ ($i=1,\dots,n$), which under local alternatives depends on $n$ through the transformation $h$. Throughout we use the notation ($i=1,\dots,n$)

[TABLE]

Further, recall the definition of $U_{i}$ in (2.8). Note that the distribution of $U_{i}$ does not depend on $n$, even under local alternatives, because $F_{Y}(Y_{i})$ is uniformly distributed on $[0,1]$, while $F_{Y}(0)=P(Y_{i}\leq 0)=P(h(Y_{i})\leq h(0))=P(S_{i}\leq 0)$ due to (2.2), and similarly $F_{Y}(1)=P(S_{i}\leq 1)$.

To formulate our main result we need some more notations. For notational convenience, define $\gamma=(c_{1},c_{2},\theta)\in\Upsilon:=C_{1}\times C_{2}\times\Theta$, which is assumed to be compact (see (A1) in Appendix A). Then, note that

[TABLE]

Further, with $Z_{i}$ from (2.8) and $S_{i}$ from (3.2) define ($i=1,\dots,n$)

[TABLE]

and let $P_{Z}$ and $F^{Z}$ denote the law and distribution function, respectively, of $Z_{i}$.

Theorem 3.1**.**

Assume (A1)–(A8) given in Appendix A. Let $(\lambda_{k})_{k\in\{1,2,\dots\}}$ be the eigenvalues of the operator

[TABLE]

with corresponding eigenfunctions $(\rho_{k})_{k\in\{1,2,\dots\}}$, which are orthonormal in the $L^{2}$-space corresponding to the distribution $F_{Z}$. Let $(W_{k})_{k\in\{1,2,\dots\}}$ be independent and standard normally distributed random variables and let $W_{0}$ be centred normally distributed with variance $E[(\tilde{\zeta}(Z_{1}))^{2}]$ such that for all $K\in\mathbb{N}$ the random vector $(W_{0},W_{1},\dots,W_{K})^{t}$ follows a multivariate normal distribution with $\operatorname{Cov}(W_{0},W_{k})=E[\tilde{\zeta}(Z_{1})\rho_{k}(Z_{1})]$ for all $k=1,\dots,K$. Then, under the local alternative $H_{1,n}$, $T_{n}$ converges in distribution to

[TABLE]

In particular, under $H_{0}$ (i.e. for $r\equiv 0$), $T_{n}$ converges in distribution to

[TABLE]

The proof is given in Appendix C. An asymptotic level-$\alpha$ test should reject $H_{0}$ if $T_{n}$ is larger than the $(1-\alpha)$-quantile of the distribution of $T$. As the distribution of $T$ depends in a complicated way on unknown quantities, we will propose a bootstrap procedure in Section 4.

Remark 3.2**.**

Note that $\zeta(z_{1},z_{2})=E[I(z_{1})I(z_{2})]$ with

[TABLE]

*Thus, the operator $K$ defined in Theorem 3.1 is positive semi-definite. *

Next we consider fixed alternatives of a transformation $h$ that do not belong to the parametric class, i. e.

[TABLE]

Theorem 3.3**.**

Assume (A1)–(A4), (A6’) given in the Appendix and let $\hat{h}$ estimate $h$ uniformly consistently on compact sets. Then, under $H_{1}$, $\lim_{n\to\infty}P(T_{n}>q)=1$ for all $q\in\mathbb{R}$, that is, the proposed test is consistent.

The proof is given in Appendix C.

The transformation model with a parametric transformation class might be useful in applications even if the model does not hold exactly. With a good choice of $\theta$ applying the transformation $\Lambda_{\theta}$ can reduce the dependence between covariates and errors enormously. Estimating an appropriate $\theta$ is much easier than estimating the transformation $h$ nonparametrically. Consequently, one might prefer the semiparametric transformation model over a completely nonparametric one. It is then of interest how far away we are from the true model. Therefore, in the following we consider testing precise hypotheses (relevant hypotheses)

[TABLE]

If a suitable test rejects $H_{0}^{\prime}$ for some small $\eta$ (fixed beforehand by the experimenter) the model is considered “good enough” to work with, even if it does not hold exactly. To test those hypotheses we will use the same test statistic as before, but we have to standardize differently. Assume $H_{0}^{\prime}$, then $h$ is a transformation which does not belong to the parametric class, i.e. the former fixed alternative $H_{1}$ holds. Let

[TABLE]

and let

[TABLE]

Note that $\underset{c_{1}\in C_{1},c_{2}\in C_{2}}{\min}\,M(\gamma)=d(\Lambda_{\theta},h)$ for all $\theta\in\Theta$. Assume that

[TABLE]

is positive definite, where $\Gamma^{\prime}_{3,3}=\dot{\Lambda}_{\theta_{0}}(Y_{1})^{t}\dot{\Lambda}_{\theta_{0}}(Y_{1})-\ddot{\Lambda}_{\theta_{0}}(Y_{1})\tilde{R}_{1}$ with

[TABLE]

and $\tilde{R}_{i}=h(Y_{i})c_{1,0}+c_{2,0}-\Lambda_{\theta_{0}}(Y_{i})$ $(i=1,\dots,n)$.

Theorem 3.4**.**

Assume (A1)–(A4), (A6’), (A8’), let (A7) hold with $\gamma_{0}$ from (A6’) and let $\Gamma^{\prime}$ be positive definite. Then

[TABLE]

with \sigma^{2}=\operatorname{Var}\big{(}w(Y_{1})\tilde{R}_{1}^{2}+\delta(Z_{1})\big{)}, where $\delta(Z_{1})=2E[w(Y_{2})\psi(Z_{1},U_{2})\tilde{R}_{2}\mid Z_{1}]$.

The proof is given in Appendix C. A consistent asymptotic level-$\alpha$-test rejects $H^{\prime}_{0}$ if $(T_{n}-n\eta)/(n\hat{\sigma}^{2})^{1/2}<u_{\alpha}$, where $u_{\alpha}$ is the $\alpha$-quantile of the standard normal distribution and $\hat{\sigma}^{2}$ is a consistent estimator for $\sigma^{2}$. Further research is required on suitable estimators for $\sigma^{2}$. For some intermediate sequence $m=m_{n}$ we considered

[TABLE]

as an estimator for $\sigma^{2}$, where $\hat{h}^{(s)}$ denotes the nonparametric estimator for $h$ depending on the subsample $(Y_{(s-1)m+1},X_{(s-1)m+1})...,(Y_{sm},X_{sm})$, but suitable choices for $m_{n}$ are still unclear.

4 A bootstrap version and simulations

Although Theorem 3.1 shows how the test statistic behaves asymptotically under $H_{0}$, it is hard to extract any information about how to choose appropriate critical values of a test that rejects $H_{0}$ for large values of $T_{n}$. The main reasons for this are that first for any function $\zeta$ the eigenvalues of the operator defined in Theorem 3.1 are unknown, that second this function is unknown and has to be estimated as well, and that third even $\psi$ (which would be needed to estimate $\zeta$) mostly is unknown and rather complex (see e.g. Appendix B). Therefore, approximating the $\alpha$-quantile, say $q_{\alpha}$, of the distribution of $T$ in Theorem 3.1 in a direct way is difficult and instead we suggest a smooth bootstrap algorithm to approximate $q_{\alpha}$.

Algorithm 4.1**.**

Let $(Y_{1},X_{1}),...,(Y_{n},X_{n})$ denote the observed data, define

[TABLE]

and let $\hat{g}$ be a consistent estimator of $g_{\theta_{0}}$, where $\theta_{0}$ is defined as in (A6) under the null hypothesis and as in (A6’) under the alternative (see Appendix A for the assumptions). Let $\kappa$ and $\ell$ be smooth Lebesgue densities on $\mathbb{R}^{d_{X}}$ and $\mathbb{R}$, respectively, where $\ell$ is strictly positive, $\kappa$ has bounded support and $\kappa(0)>0$. Let $(a_{n})_{n}$ and $(b_{n})_{n}$ be positive sequences with $a_{n}\to 0$, $b_{n}\to 0$, $na_{n}\to\infty$, $nb_{n}^{d_{X}}\to\infty$. Denote by $m\in\mathbb{N}$ the sample size of the bootstrap sample.

1. (1)

Calculate $\hat{\gamma}=(\hat{c}_{1},\hat{c}_{2},\hat{\theta})^{t}=\arg\underset{\gamma\in\Upsilon}{\min}\,\sum_{i=1}^{n}w(Y_{i})(\hat{h}(Y_{i})c_{1}+c_{2}-\Lambda_{\theta}(Y_{i}))^{2}$. Estimate the parametric residuals $\varepsilon_{i}(\theta_{0})=h_{\theta_{0}}(Y_{i})-g_{\theta_{0}}(X_{i})$ by $\hat{\varepsilon}_{i}=h_{\hat{\theta}}(Y_{i})-\hat{g}(X_{i})$ and denote centered versions by $\tilde{\varepsilon}_{i}=\hat{\varepsilon}_{i}-n^{-1}\sum_{j=1}^{n}\hat{\varepsilon}_{j}$, $i=1,\dots,n$. 2. (2)

Generate $X_{j}^{*}$, $j=1,\dots,m$, independently (given the original data) from the density

[TABLE]

(which is a kernel density estimator for $f_{X}$ with kernel $\kappa$ and bandwidth $b_{n}$). For $j=1,\dots,m$ define bootstrap observations as

[TABLE]

where $\varepsilon_{j}^{*}$ is generated independently (given the original data) from the density

[TABLE]

(which is a kernel density estimator for the density of $\varepsilon(\theta_{0})$ with kernel $\ell$ and bandwidth $a_{n}$). 3. (3)

Calculate the bootstrap estimate $\hat{h}^{*}$ for $h^{*}$ from $(Y_{j}^{*},X_{j}^{*}),j=1,...,m$. 4. (4)

Calculate the bootstrap statistic $T_{n,m}^{*}=\underset{(c_{1},c_{2},\theta)\in\Upsilon}{\min}\,\sum_{j=1}^{m}w(Y_{j}^{*})(\hat{h}^{*}(Y_{j}^{*})c_{1}+c_{2}-\Lambda_{\theta}(Y_{j}^{*}))^{2}$. 5. (5)

Let $B\in\mathbb{N}$. Repeat steps (2)–(4) $B$ times to obtain the bootstrap statistics $T_{n,m,1}^{*},\dots,T_{n,m,B}^{*}$. Let $q_{\alpha}^{*}$ denote the quantile of $T_{n,m}^{*}$ conditional on $(Y_{i},X_{i}),i=1,...,n$. Estimate $q_{\alpha}^{*}$ by

[TABLE]

Remark 4.2**.**

The properties $nb_{n}^{d_{X}}\rightarrow\infty$ and $\kappa(0)>0$ ensure that conditional on the original data $(Y_{1},X_{1}),...,(Y_{n},X_{n})$ the support of $X^{*}$ contains that of $v$ (from assumption (B7) in Appendix B) with probability converging to one. Thus, $v$ can be used for calculating $\hat{h}^{*}$ as well. 2. 2.

To proceed as in Algorithm 4.1 it may be necessary to modify $h^{*}$ so that $S_{j}^{*}=\hat{g}(X_{j}^{*})+\varepsilon_{j}^{*}$ belongs to the domain of $(h^{*})^{-1}$ for all $j=1,...,m$. As long as these modifications do not have any influence on $h^{*}(y)$ for $y\in\mathcal{Y}_{w}$, the influence on the $\hat{h}^{*}$ and $T_{n,m}$ should be asymptotically negligible (which can be proven for the estimator by Colling and Van Keilegom (2019)**).

The bootstrap algorithm should fulfil two properties: On the one hand, under the null hypothesis the algorithm has to provide, conditionally on the original data, consistent estimates of the quantiles of $T_{n}$, or to be precise its asymptotic distribution from Theorem 3.1. To formalize this, let $(\Omega,\mathcal{A},P)$ denote the underlying probability space. Assume that $(\Omega,\mathcal{A})$ can be written as $\Omega=\Omega_{1}\times\Omega_{2}$ and $\mathcal{A}=\mathcal{A}_{1}\otimes\mathcal{A}_{2}$ for some measurable spaces $(\Omega_{1},\mathcal{A}_{1})$ and $(\Omega_{2},\mathcal{A}_{2})$. Further, assume that $P$ is characterized as the product of a probability measure $P_{1}$ on $(\Omega_{1},\mathcal{A}_{1})$ and a Markov kernel

[TABLE]

that is $P=P_{1}\otimes P_{2}^{1}$. While randomness with respect to the original data is modelled by $P_{1}$, randomness with respect to the bootstrap data and conditional on the original data is modelled by $P_{2}^{1}$. Moreover, assume

[TABLE]

With these notations in mind for all $q\in(0,\infty)$ it would be desirable to obtain

[TABLE]

for all $\delta>0$ and $n\rightarrow\infty$. Here, the convention

[TABLE]

is used. On the other hand, to be consistent under $H_{1}$ the bootstrap quantiles have to stabilize or at least converge to infinity with a rate less than that of $T_{n}$. To be precise, it is needed that

[TABLE]

for all $\delta>0$.

In the supplement we give conditions under which the bootstrap Algorithm 4.1 has the desired properties (4.2) and (4.3). In particular we need an expansion of $\hat{h}^{*}$ as bootstrap counterpart to (2.7). To formulate this, for any realisation $w\in\Omega_{1}$ define

[TABLE]

Then for any compact set $\mathcal{K}\subseteq\mathbb{R}$ and

[TABLE]

we need

[TABLE]

for $n\rightarrow\infty$, where $\psi^{*}$ fulfils some assumptions given in the supplement (see assumption (A8*) for details). In the supplement we also give conditions under which for the transformation estimator of Colling and Van Keilegom (2019) the expansion is valid (see Lemma D.8).

Simulations

Throughout this section, $g(X)=4X-1$, $X\sim\mathcal{U}([0,1])$ and $\varepsilon\sim\mathcal{N}(0,1)$ are chosen. Moreover, the null hypothesis of $h$ belonging to the Yeo and Johnson (2000) transformations

[TABLE]

with parameter $\theta\in\Theta_{0}=[0,2]$ is tested. Under $H_{0}$ we generate data using the transformation $h=(\Lambda_{\theta_{0}}(\cdot)-\Lambda_{\theta_{0}}(0))/(\Lambda_{\theta_{0}}(1)-\Lambda_{\theta_{0}}(0))$ to match the identification constraints $h(0)=0,h(1)=1$. Under the alternative we choose transformations $h$ with an inverse given by the following convex combination,

[TABLE]

for some $\theta_{0}\in[0,2]$, some strictly increasing function $r$ and some $c\in[0,1]$. In general it is not clear if a growing factor $c$ leads to a growing distance (2.5). Indeed, the opposite might be the case, if $r$ is somehow close to the class of transformation functions considered in the null hypothesis. Simulations were conducted for $r_{1}(Y)=5\Phi(Y)$, $r_{2}(Y)=\exp(Y)$ and $r_{3}(Y)=Y^{3}$, where $\Phi$ denotes the cumulative distribution function of a standard normal distribution, and $c=0,0.2,0.4,0.6,0.8,1$. The prefactor in the definition of $r_{1}$ is introduced because the values of $\Phi$ are rather small compared to the values of $\Lambda_{\theta}$, that is, even when using the presented convex combination in (4.5), $\Lambda_{\theta_{0}}$ (except for $c=1$) would dominate the “alternative part” $r$ of the transformation function without this factor. Note that $r_{2}$ and $\Lambda_{0}$ only differ with respect to a different standardization. Therefore, if $h$ is defined via (4.5) with $r=r_{2}$ the resulting function is for $c=1$ close to the null hypothesis case.

For calculating the test statistic the weighting function $w$ was set equal to one. The nonparametric estimator of $h$ was calculated as in Colling and Van Keilegom (2019) (see Appendix B for details) with the Epanechnikov kernel $K(y)=\frac{3}{4}(1-y^{2})^{2}I_{[-1,1]}(y)$ and a normal reference rule bandwidth (see for example Silverman (1986))

[TABLE]

where $\hat{\sigma}_{u}^{2}$ and $\hat{\sigma}_{x}^{2}$ are estimators for the variance of $U=\mathcal{T}(Y)$ and $X$, respectively. The number of evaluation points $N_{x}$ for the nonparametric estimator of $h$ was set equal to $100$ (see Appendix B for details). The integral in (B.2) was computed by applying the function integrate implemented in R. In each simulation run $n=100$ independent and identically distributed random pairs $(Y_{1},X_{1}),...,(Y_{n},X_{n})$ were generated as described before and $250$ bootstrap quantiles, which are based on $m=100$ bootstrap observations $(Y_{1}^{*},X_{1}^{*}),...,(Y_{m}^{*},X_{m}^{*})$, were calculated as in Algorithm 4.1 using $\kappa$ the $U([-1,1])$-density, $\ell$ the standard normal density and $a_{n}=b_{n}=0.1$. To obtain more precise estimators of the rejection probabilities under the null hypothesis, $800$ simulation runs were performed for each choice of $\theta_{0}$ under the null hypothesis, whereas in the remaining alternative cases $200$ runs were conducted. Among other things the nonparametric estimation of $h$, the integration in (B.2), the optimization with respect to $\theta$ and the number of bootstrap repetitions cause the simulations to be quite computationally demanding. Hence, an interface for C++ as well as parallelization were used to conduct the simulations.

The main results of the simulation study are presented in Table 1. There, the rejection probabilities of the settings with $h=(\Lambda_{\theta_{0}}(\cdot)-\Lambda_{\theta_{0}}(0))/(\Lambda_{\theta_{0}}(1)-\Lambda_{\theta_{0}}(0))$ under the null hypothesis, and $h$ as in (4.5) under the alternative with $r\in\{r_{1},r_{2},r_{3}\}$, $c\in\{0,0.2,0.4,0.6,0.8,1\}$ and $\theta_{0}\in\{0,0.5,1,2\}$ are listed. The significance level was set equal to 0.05 and 0.10. Note that the test sticks to the level or is even a bit conservative. Under the alternatives the rejection probabilities not only differ between different choices of $r$, but also between different transformation parameters $\theta_{0}$ that are inserted in (4.5). While the test shows high power for some alternatives, there are also cases, where the rejection probabilities are extremely small. There are certain reasons that explain these observations. First, the class of Yeo-Johnson transforms seems to be quite general and second the testing approach itself is rather flexible due to the minimization with respect to $\gamma$. Having a look at the definition of the test statistic in (2.6), it attains small values if the true transformation function can be approximated by a linear transformation of $\Lambda_{\tilde{\theta}}$ for some appropriate $\tilde{\theta}\in[0,2]$. In the following, this issue will be explored further by analysing some graphics. All of the figures that occur in the following have the same structure and consist of four panels. The upper left panel shows the true transformation function with inverse function (4.5). Due to the choice of $g(X)=4X-1$ and $X\sim\mathcal{U}([0,1])$ the vertical axis reaches from $-1$ to 3, which would be the support of $h(Y)$ if the error is neglected. In the upper right panel the parametric estimator of this function is displayed. Both of these functions are then plotted against each other in the lower left panel. Finally, the function $Y\mapsto\Lambda_{\theta_{0}}(Y(\Lambda_{\theta_{0}}^{-1}(1)-\Lambda_{\theta_{0}}^{-1}(0))+\Lambda_{\theta_{0}}^{-1}(0))$, which represents the part of $h$ corresponding to the null hypothesis, is shown in the last panel.

In the lower left panel one can see if the true transformation function can be approximated by a linear transform of some $\Lambda_{\tilde{\theta}},\tilde{\theta}\in[0,2]$, which is an indicator for rejecting or not rejecting the null hypothesis as was pointed out before. As already mentioned, the rejection probabilities not only differ between different deviation functions $r$, but also within these settings. For example, when considering $r=r_{1}$ with $c=0.6$ the rejection probabilities for $\theta_{0}=0.5$ amount to $0.035$ for $\alpha=0.05$ and to $0.050$ for $\alpha=0.10$, while for $\theta_{0}=2$ they are $0.415$ and $0.545$. Figures 1 and 2 explain why the rejection probabilities differ that much. While for $\theta_{0}=0.5$ the transformation function can be approximated quite well by transforming $\Lambda_{1.06}$ linearly, the best approximation for $\theta_{0}=2$ is given by $\Lambda_{1.94}$ and seems to be relatively bad. The best approximation for $c=1$ can be reached for $\theta$ around $1.4$. In contrast to that, considering $\theta_{0}=2$ and $r=r_{3}$ results in a completely different picture. As can be seen in Figure 3 even for $c=0.2$ the resulting $h$ differs so much from the null hypothesis that it can not be linearly transformed into a Yeo-Johnson transform (see the lower left subgraphic). Consequently, the rejection probabilities are rather high.

Under some alternatives the rejection probabilities are even smaller than the level. This behaviour indicates that from the presented test’s perspective these models seem to fulfil the null hypothesis more convincingly than the null hypothesis models themselves. The reason for this can be seen in Figure 4 for the setting $\theta_{0}=1,c=0.4$ and $r=r_{1}$. There, the relationship between the nonparametric estimator of the transformation function and the true transformation function is shown. While the diagonal line represents the identity, the nonparametric estimator seems to flatten the edges of the transformation function. In contrast to this, using $r=r_{1}$ in (4.5) steepens the edges so that both effects neutralize each other. Similar effects cause low rejection probabilities for $r=r_{2}$, although the reasoning is slightly more sophisticated and is also associated with the boundedness of the parameter space $\Theta_{0}=[0,2]$.

One possible solution could consist in adjusting the weight function $w$ such that the boundary of the support of $Y$ does no longer belong to the support of $w$. In Table 2 the rejection probabilities for a modified weighting approach are presented. There, the weight function was chosen such that the smallest five percent and the largest five percent of observations were omitted to avoid the flattening effect of the nonparametric estimation. Indeed, the resulting rejection probabilities under the alternatives increase and lie above those under the null hypotheses.

Acknowledgements. Natalie Neumeyer acknowledges financial support by the DFG (Research Unit FOR 1735 Structural Inference in Statistics: Adaptation and Efficiency). Ingrid Van Keilegom acknowledges financial support by the European Research Council (2016- 2021, Horizon 2020 / ERC grant agreement No. 694409).

Appendix A Assumptions for the main results

In the following assumptions let $\mathcal{Y}$ denote the support of $Y$ (which depends on $n$ under local alternatives). Further, $F_{S}$ denotes the distribution function of $S_{1}$ as in (3.2) and $\mathcal{T}_{S}$ denotes the transformation $s\mapsto(F_{S}(s)-F_{S}(0))/(F_{S}(1)-F_{S}(0))$.

1. (A1)

The sets $C_{1},C_{2}$ and $\Theta$ are compact. 2. (A2)

The weight function $w$ is continuous with a compact support $\mathcal{Y}_{w}\subset\mathcal{Y}$. 3. (A3)

The map $(y,\theta)\mapsto\Lambda_{\theta}(y)$ is twice continuously differentiable on $\mathcal{Y}_{w}$ with respect to $\theta$ and the (partial) derivatives are continuous in $(y,\theta)\in\mathcal{Y}_{w}\times\Theta$. 4. (A4)

There exists a unique strictly increasing and continuous transformation $h$ such that model (2.1) holds with $X$ independent of $\varepsilon$. 5. (A5)

The function $h_{0}$ defined in (3.1) is strictly increasing and continuously differentiable and $r$ is continuous on $\mathcal{Y}_{w}$. $F_{Y}$ is strictly increasing on the support of $Y$. 6. (A6)

Minimizing the function M:\Upsilon\rightarrow\mathbb{R},\gamma=(c_{1},c_{2},\theta)\mapsto E\big{[}w(Y)(h_{0}(Y)c_{1}+c_{2}-\Lambda_{\theta}(Y))^{2}\big{]} leads to a unique solution $\gamma_{0}=(c_{1,0},c_{2,0},\theta_{0})$ in the interior of $\Upsilon$. For all $\theta\neq\tilde{\theta}$ it is \underset{y\in\operatorname{supp}(w)}{\sup}\,\big{|}\frac{\Lambda_{\theta}(y)-\Lambda_{\theta}(0)}{\Lambda_{\theta}(1)-\Lambda_{\theta}(0)}-\frac{\Lambda_{\tilde{\theta}}(y)-\Lambda_{\tilde{\theta}}(0)}{\Lambda_{\tilde{\theta}}(1)-\Lambda_{\tilde{\theta}}(0)}\big{|}>0. 7. (A7)

The Hessian matrix $\Gamma:=\operatorname{Hess}M(\gamma_{0})$ is positive definite. 8. (A8)

The transformation estimator $\hat{h}$ fulfills (2.7) for some function $\psi$. For some $\mathcal{U}_{0}$ (independent of $n$ under local alternatives) with $\mathcal{T}_{S}(h(\mathcal{Y}_{w}))\subset\mathcal{U}_{0}$ the function class $\{z\mapsto\psi(z,t):t\in\mathcal{U}_{0}\}$ is Donsker with respect to $P^{Z}$ and $E[\psi(Z_{1},t)]=0$ for all $t\in\mathcal{U}_{0}$. The fourth moment $E[w(h_{0}^{-1}(S_{1}))\psi(Z_{1},U_{1})^{4}]$ is finite and the conditional moments $E[w(h_{0}^{-1}(S_{1}))\psi(Z_{1},U_{2})^{2}|Z_{1}=z]$ are locally bounded.

When considering a fixed alternative $H_{1}$ or the relevant hypothesis $H^{\prime}_{0}$, (A6) and (A8) are replaced by the following Assumptions (A6’) and (A8’) (assumption (A8’) is only relevant for $H^{\prime}_{0}$). Note that $h$ is a fixed function then, not depending on $n$.

1. (A6’)

Minimizing the function M:\Upsilon\rightarrow\mathbb{R},\gamma=(c_{1},c_{2},\theta)\mapsto E\big{[}w(Y)(h(Y)c_{1}+c_{2}-\Lambda_{\theta}(Y))^{2}\big{]} leads to a unique solution $\gamma_{0}=(c_{1,0},c_{2,0},\theta_{0})$ in the interior of $\Upsilon$. For all $\theta\neq\tilde{\theta}$ it is \underset{y\in\operatorname{supp}(w)}{\sup}\,\big{|}\frac{\Lambda_{\theta}(y)-\Lambda_{\theta}(0)}{\Lambda_{\theta}(1)-\Lambda_{\theta}(0)}-\frac{\Lambda_{\tilde{\theta}}(y)-\Lambda_{\tilde{\theta}}(0)}{\Lambda_{\tilde{\theta}}(1)-\Lambda_{\tilde{\theta}}(0)}\big{|}>0.

1. (A8’)

The transformation estimator $\hat{h}$ fulfills (2.7) for some function $\psi$. For some $\mathcal{U}_{0}\supset\mathcal{T}_{S}(h(\mathcal{Y}_{w}))$ the function class $\{z\mapsto\psi(z,t):t\in\mathcal{U}_{0}\}$ is Donsker with respect to $P^{Z}$ and $E[\psi(Z_{1},t)]=0$ for all $t\in\mathcal{U}_{0}$. Further, one has $E[\psi(Z_{1},U_{2})^{2}]<\infty$.

Appendix B Nonparametric transformation estimation

In this section we consider a transformation estimator which fulfills assumption (A8) and in particular the expansion (2.7). To this end we reproduce the definitions of Colling and Van Keilegom (2019) and prove Lemma B.2 below. Denote the conditional distribution function of $U_{1}$ from (2.8), given $X_{1}=x$, by $F_{U|X}(\cdot|x)$ and estimate it by

[TABLE]

Here $h_{x}$ and $h_{u}$ are bandwidths and $K$ is an appropriate kernel function (as in assumptions (B4) and (B5) below),

[TABLE]

Further, consider some kernel $L$ and bandwidth $b\searrow 0$ fulfilling assumption (B6) and define

[TABLE]

where L_{b}(\cdot)=\frac{1}{b}L\big{(}\frac{\cdot}{b}\big{)} and

[TABLE]

Remark B.1**.**

In principle, the derivative with respect to any other component $x_{i}$ fulfilling assumption (B3) below can be used in (B2) as well (similar to Chiappori et al. (2015)). W.l.o.g. only the case $i=1$ is considered here.

Let $\hat{F}_{Y}$ be the empirical distribution function of $Y_{1},\dots,Y_{n}$ and define

[TABLE]

and estimate $U_{i}$ from (2.8) by $\hat{U}_{i}:=\hat{\mathcal{T}}(Y_{i})$. The estimator for $h$ can be defined as

[TABLE]

According to Colling and Van Keilegom (2019) (see their proof of Propositions 6 and 7) one has expansion (2.7) for $\hat{h}$ under the null hypothesis of the parametric transformation class. For the definition of $\psi(Z_{i},u)$ from (2.7) in (B.6) below we need the following notations. Let $f_{U,X}$ denote the joint density of $(U,X)$ and define

[TABLE]

Then, the conditional distribution function of $U$ conditioned on $X=x$ can be written as

[TABLE]

Further, let

[TABLE]

Define $\tilde{v}_{1}(u_{0},x)=\frac{v(x)}{s_{1}(u_{0},x)}$, $\tilde{v}_{2}(u_{0},x)=\frac{v(x)s_{1}(u_{0},x)}{s_{1}(1,x)^{2}}$ and (for $\tilde{v}\in\{\tilde{v}_{1},\tilde{v}_{2}\}$)

[TABLE]

see Colling and Van Keilegom (2019) for details. Then, with $Q(\cdot)=h(\mathcal{T}^{-1}(\cdot))$ the function $\psi$ in the expansion (2.7) can be written as

[TABLE]

Note that $E[\psi(Z_{j},u)]=0$ for all $u$.

In the following assumptions (adjusted from Colling and Van Keilegom (2019)) are given which ensure (A8) for the estimator $\hat{h}$ from (B.4). Let, as in (A8), $\mathcal{T}_{S}(h(\mathcal{Y}_{w}))\subset\mathcal{U}_{0}$, where $\mathcal{U}_{0}$ is independent from $n$ under local alternatives and lies in the interior of the support of $U$. Let $\mathcal{X}\subset\mathbb{R}^{d_{X}}$ denote the support of $X$.

1. (B1)

The cumulative distribution function of $\varepsilon$ is absolutely continuous and has a density that is continuous on its support. Furthermore, $X$ and $\varepsilon$ are independent and $\mathcal{U}_{0}$ is a connected subset of $\mathbb{R}$. 2. (B2)

The transformation $Q(\cdot)=h(\mathcal{T}^{-1}(\cdot))$ is strictly increasing and continuously differentiable on $\mathcal{U}_{0}$. 3. (B3)

The set

[TABLE]

is nonempty. 4. (B4)

The bandwidths $h_{x}$ and $h_{u}$ satisfy for an appropriate $q\in\mathbb{N}$

[TABLE] 5. (B5)

The kernel $K$ is symmetric with a connected and compact support containing some neighbourhood around 0. Further, $K$ is $q$-times continuously differentiable with $K$ and $K^{\prime}$ being of bounded variation. Moreover, $\int K(z)\,dz=1,\int z^{l}K(z)\,dz=0$ for all $l=1,...,q-1$. 6. (B6)

The kernel $L$ is twice continuously differentiable with uniformly bounded derivatives and with median 0, and $b=b_{n}>0$ is a bandwidth sequence that satisfies $nb^{4}\rightarrow\infty$ and $b\sqrt{n}h_{x}^{d_{X}}(\min(h_{x},h_{u}))^{2}/\log(n)\rightarrow\infty$. 7. (B7)

$v$ is a weight function with compact support $\mathcal{X}_{0}\subset\mathcal{X}_{\partial 1}$ with nonempty interior. Further, $\int_{\mathcal{X}_{0}}v(x)\,dx=1$ and $v$ is $q$-times continuously differentiable and all these derivatives are uniformly bounded in the interior, i.e.,

[TABLE]

for all $\alpha_{1},...,\alpha_{d_{X}}\in\{0,...,q-1\}$ with $|\alpha|=\sum_{i=1}^{d_{X}}\alpha_{i}\leq m$. 8. (B8)

The regression function $g$ is continuously differentiable with respect to $x_{i}$ on $\mathcal{X}$ for $i=1,...,d_{X}$. 9. (B9)

The joint density function $f_{Y,X}(y,x)$ of $(Y,X)$ is uniformly bounded, $(q+2)$-times continuously differentiable and all these derivatives are uniformly bounded, i.e.,

[TABLE]

for all $\alpha_{0},\alpha_{1},...,\alpha_{d_{X}}\in\{0,...,q-1\}$ with $|\alpha|=\sum_{i=0}^{d_{X}}\alpha_{i}\leq q+2$. Further, we assume $\underset{y:\,\mathcal{T}(y)\in\mathcal{U}_{0}}{\inf}\,f_{Y}(y)>0$, where $f_{Y}$ is the density function of $Y$. 10. (B10)

Assume

[TABLE]

The following result holds under the null hypothesis $H_{0}$, under fixed alternatives $H_{1}$ and under local alternatives $H_{1,n}$.

Lemma B.2**.**

Assume (A4) and (B1)–(B10). Then, assumption (A8) is fulfilled and especially the expansion (2.7) holds for $\hat{h}$ with $\psi$ from (B.6).

Proof. In the case of a fixed transformation $h$, the assertion is covered by Theorems 5.1 and 5.2 in Colling and Van Keilegom (2019). Therefore, only local alternatives need to be considered. To this end we consider the transformation class

[TABLE]

for an appropriate set $A$. In case of local alternatives as in equation (3.1), consider for example $A=\{n^{-\frac{1}{2}}:n\in\mathbb{N},n\geq N\}$ for a sufficiently large $N\in\mathbb{N}$. The expansion is shown uniformly in $\alpha\in A$, that is uniformly in $h\in\mathcal{H}$, and uniformly in $y\in\mathcal{Y}_{w}$. Nevertheless, most arguments used for fixed $h$ as in Colling and Van Keilegom (2019) are still valid. Note that in our framework $S=g(X)+\varepsilon$ does not depend on $n$, and consider $Y=h^{-1}(S)$ for $h\in\mathcal{H}$. First, note that neither the $U_{i}$ nor the $\hat{U}_{i}$ depend on $\alpha$, since $\mathds{1}_{\{Y_{j}\leq Y_{i}\}}=\mathds{1}_{\{h(Y_{j})\leq h(Y_{i})\}}$ is independent of $\alpha$ and $h(0)=0$ as well as $h(1)=1$ for all $h\in\mathcal{H}$. Hence, $Q$ from (B2) and its estimator $\hat{Q}$ from (B.1) are independent of $\alpha\in A$. Moreover, $\hat{Q}^{\prime}$ is uniformly consistent. By standard arguments it can be shown that for $\hat{\mathcal{T}}$ from (B.3) and $\mathcal{T}$ from (2.8) one has

[TABLE]

uniformly in $y\in\mathcal{Y}_{w}$ and $\alpha\in A$, so that

[TABLE]

$\Box$

Appendix C Proofs of the main results

Proof of Theorem 3.1. For ease of presentation define

[TABLE]

such that $T_{n}=\min_{\gamma\in\Upsilon}M_{n}(\gamma)$. Let $\hat{\gamma}=(\hat{c_{1}},\hat{c_{2}},\tilde{\theta})$ denote the minimizer of $M_{n}$ and $\gamma_{0}$ be the vector such that

[TABLE]

(see assumption (A6)). Note that $c_{1,0}=\Lambda_{\theta_{0}}(1)-\Lambda_{\theta_{0}}(0)$, $c_{2,0}=\Lambda_{\theta_{0}}(0)$.

We have $\tilde{\gamma}-\gamma_{0}=o_{P}(1)$ because for all $\gamma=(c_{1},c_{2},\theta)$ in a compact set that does not contain $\gamma_{0}$ and an appropriate $\varepsilon>0$ one has

[TABLE]

uniformly in $\gamma\in\Upsilon$ (see the proof of Theorem 3.3 for details).

Let in the following $\nabla f$ denote the gradient of a function $f$ and $\operatorname{Hess}f$ denote the Hessian matrix of $f$. Note that $\nabla M_{n}(\tilde{\gamma})=0$ and thus by Taylor expansion

[TABLE]

where $\gamma^{*}=(c_{1}^{*},c_{2}^{*},\theta^{*})$ is on the line between $\gamma_{0}$ and $\hat{\gamma}$. Further, with $\hat{\Gamma}_{3,3}=\dot{\Lambda}_{{\theta^{*}}}(Y_{k})^{t}\dot{\Lambda}_{{\theta^{*}}}(Y_{k})-\ddot{\Lambda}_{{\theta^{*}}}(Y_{k})(\hat{h}(Y_{k}){c}_{1}^{*}+{c}_{2}^{*}-\Lambda_{{\theta^{*}}}(Y_{k}))$ we have

[TABLE]

To obtain the last equality note that $\hat{h}$ converges to $h_{0}$ uniformly on $\mathcal{Y}_{w}$ thanks to (2.7) under local alternatives. Further, $h^{-1}$ converges to $h_{0}^{-1}$ uniformly on compacta and, as $\tilde{\gamma}$ converges to $\gamma_{0}$, $\dot{\Lambda}_{\tilde{\theta}}$ and $\ddot{\Lambda}_{\tilde{\theta}}$ converge to $\dot{\Lambda}_{\theta_{0}}$ and $\ddot{\Lambda}_{\theta_{0}}$, respectively, uniformly on $\mathcal{Y}_{w}$. To obtain (C.8) it remains to apply the law of large numbers and (C.2).

Since $\Gamma_{0}$ is positive definite by assumption and $M_{n}(\gamma_{0})$ and $M_{n}(\hat{\gamma})$ are bounded in probability, one obtains from (C.3) that $\hat{\gamma}-\gamma_{0}=O_{P}(n^{-1/2})$.

Now, again by Taylor expansion, for all values $\gamma$ with $\gamma-\gamma_{0}=O_{P}(n^{-1/2})$,

[TABLE]

where the map $q$ is defined via

[TABLE]

with $R(\cdot)$ from (3.3). The minimizer $z_{0}$ of the quadratic function $q$ can easily be obtained as

[TABLE]

where we have inserted the expansion from (2.7) as well as (3.1) and use the definition $\beta=E[w(h_{0}^{-1}(S_{1}))r_{0}(h_{0}^{-1}(S_{1}))R(S_{1})]$. It is also easy to see that (for some $\bar{\gamma}$)

[TABLE]

so that it is sufficient to consider $q(z_{0})=\min_{z}q(z)$ instead of $T_{n}$.

Inserting the expansion for $z_{0}$ into $q(\cdot)$ as well as inserting the expansion for $\hat{h}$ from (2.7) and (3.1) into $M_{n}(\gamma_{0})$ gives

[TABLE]

With some simple calculations of variances one shows that, after centering the multiple sums, those terms are negligible, where some of the indices coincide. Considering the (centred) multiple sums with distinct indices only, for the resulting U-statistics Hoeffding decompositions are applied (see, e.g. Section 1.6 in (Lee, 1990)). Again with simple, but tedious calculations of variances one obtains the following dominating terms,

[TABLE]

where $U_{n}$ is a U-statistic of order 2, i.e.

[TABLE]

with degenerate kernel

[TABLE]

which coincides with $\zeta(z_{1},z_{2})$ from (3.6). Further $W_{0,n}=n^{-1}\sum_{i=1}^{n}\tilde{\zeta}(Z_{i})$ with

[TABLE]

which coincides with $\tilde{\zeta}(z)$ from (3.8). Furthermore

[TABLE]

with $\bar{r}$ from (3.7).

Note that $\zeta$ is symmetric. Hence, referring to Witting and Müller-Funk (1995, p. 141) it can be written as

[TABLE]

(in $L^{2}$ sense corresponding to the distribution $F_{Z}$) with notations from Theorem 3.1. Referring to Remark 3.2 $\zeta$ is positive semi-definite, which results in $\lambda_{k}\geq 0,k\in\mathbb{N}$. From classical results on U-statistics, $nU_{n}$ converges to $\sum_{k=1}^{\infty}\lambda_{k}(W_{k}^{2}-1)$ in distribution (again with notations as in Theorem 3.1), see e.g. Lee (1990), Theorem 1 in Section 3.2.2. On the other hand, $n^{1/2}W_{0,n}$ converges to a normal distribution by the central limit theorem. As $U_{n}$ and $W_{0,n}$ are dependent, we have to go through some of the steps of the proof of Theorem 1 in Lee (1990, p. 79) to obtain the limiting distribution of $T_{n}$. Lee (1990) uses the truncated sums $\sum_{k=1}^{K}\lambda_{k}\rho_{k}(Z_{i})\rho_{k}(Z_{j})\approx\zeta(Z_{i},Z_{j})$ (for large $K$) to obtain the approximation

[TABLE]

with $W_{k,n}=n^{-1/2}\sum_{i=1}^{n}\rho_{k}(Z_{i})$ and $v_{k,n}=n^{-1}\sum_{i=1}^{n}\rho_{k}^{2}(Z_{i})=1+o_{P}(1)$ by the law of large numbers and the orthonormality of the eigenfunctions. Now to obtain convergence of $nU_{n}+n^{1/2}W_{0,n}$, note that applying the multivariate central limit theorem, $(W_{0,n},W_{1,n},\dots,W_{K,n})^{t}$ converges in distribution to $(W_{0},W_{1},\dots,W_{K})^{t}$ as defined in Theorem 3.1, for each $K$. Hence, by the continuous mapping theorem we obtain

[TABLE]

for each $K$. Proceeding as in the proof of Theorem 1 in Lee (1990, p. 79) by letting $K\to\infty$, one obtains $\sum_{k=1}^{\infty}\lambda_{k}(W_{k}^{2}-1)+W_{0}$ as limit of $nU_{n}+n^{1/2}W_{0,n}$. Note further that (C.10) especially leads to

[TABLE]

such that $nU_{n}+b+n^{1/2}W_{0,n}$ converges to $\sum_{k=1}^{\infty}\lambda_{k}W_{k}^{2}+W_{0}$, which completes the proof of Theorem 3.1. $\Box$

Proof of Theorem 3.3. Note that the functions $f_{\gamma}(y)=w(y)(h(y)c_{1}+c_{2}-\Lambda_{\theta}(y))^{2}$ are bounded, the parameter set $C_{1}\times C_{2}\times\Theta$ is compact and for every $y$ the map $\gamma=(c_{1},c_{2},\theta)\mapsto w(y)(h(y)c_{1}+c_{2}-\Lambda_{\theta}(y))^{2}$ is continuous. Hence, following Lemma 6.1 in (Wellner, 2005) the class $\mathcal{F}=\{f_{\gamma}:\gamma\in C_{1}\times C_{2}\times\Theta\}$ is a Glivenko-Cantelli class. This leads to

[TABLE]

for some $\gamma^{*}=(c_{1}^{*},c_{2}^{*},\theta^{*})^{t}\in\Upsilon$ (remind that $\Upsilon$ is compact). Under the fixed alternative $H_{1}$ one has $E[w(Y)(h(Y)c_{1}^{*}+c_{2}^{*}-\Lambda_{\theta^{*}}(Y))^{2}]>0$, so that $T_{n}\to\infty$. $\Box$

Proof of Theorem 3.4. The beginning of the proof is similar to the proof of Theorem 3.1 and we will state the main differences. Again, we write $T_{n}=\min_{\gamma\in\Upsilon}M_{n}(\gamma)=M_{n}(\hat{\gamma})$ with $M_{n}$ as in (C.1). Recall that by (A6), $\gamma_{0}=(c_{0,1},c_{0,2},\theta_{0})$ is the unique minimizer of $M(c_{1},c_{2},\theta)=E[w(Y_{1})(h(Y_{1})c_{1}+c_{2}-\Lambda_{\theta}(Y_{1}))^{2}]$. We can again derive $\hat{\gamma}-\gamma_{0}=o_{P}(1)$ and further (C.3) and (C.8), but now with $\Gamma_{0}$ replaced by $\Gamma^{\prime}$ as in (3.9), which is the limit of $n^{-1}\operatorname{Hess}M_{n}(\gamma_{0})$. However, $M_{n}(\gamma_{0})$ and $M_{n}(\hat{\gamma})$ are not bounded in probability under the model considered here. Instead we will show that

[TABLE]

and thus we can derive from (C.3) that $\hat{\gamma}-\gamma_{0}=O_{P}(n^{-1/4})$. To obtain (C.11), define

[TABLE]

and let $\tilde{M}_{n}(\gamma^{*})=\min_{\gamma\in\Upsilon}\tilde{M}_{n}(\gamma)$. Note that $E[M_{n}(\gamma)]=nM(\gamma)$. Now one can obtain

[TABLE]

where for (C.12) one applies (2.7), whereas (C.13) holds because the empirical process $n^{-1/2}(\tilde{M}_{n}(\gamma)-nM(\gamma))$, $\gamma\in\Upsilon$, is Donsker. Because

[TABLE]

by (C.12), to show (C.11) it is sufficient to show

[TABLE]

To derive (C.14) note that

[TABLE]

by (C.12) and (C.13). On the other hand

[TABLE]

by (C.12) and (C.13). Both inequalities together imply (C.14) and consequently (C.11) holds.

Again similar to the proof of Theorem 3.1 we obtain by Taylor expansion that, for all values $\gamma$ with $\gamma-\gamma_{0}=O_{P}(n^{-1/4})$,

[TABLE]

where the map $q$ is defined via

[TABLE]

with $\Gamma^{\prime}$ from (3.9) and $\hat{R}_{k}=(\hat{h}(Y_{k}),1,-\dot{\Lambda}_{\theta_{0}}(Y_{k}))^{t}$ ($k=1,\dots,n$). The minimizer $z_{0}$ of the quadratic function $q$ can easily be obtained as

[TABLE]

with $R_{k}=(h(Y_{k}),1,-\dot{\Lambda}_{\theta_{0}}(Y_{k}))^{t}$. To obtain the rate note that $E[w(Y_{k})(h(Y_{k})c_{1,0}+c_{2,0}-\Lambda_{\theta_{0}}(Y_{k}))R_{k}]=0$ because $\gamma_{0}$ minimizes $M(\gamma)$ and thus one has $\nabla M(\gamma_{0})=0$. As in the proof of Theorem 3.1 it follows that instead of considering $T_{n}/n$ one can consider $q(z_{0})/n$ to derive the limiting distribution. To this end note that

[TABLE]

The first term on the right hand side can be treated as in the proof of Theorem 3.1 by inserting the expansion from (2.7) to obtain the rate $o_{P}(n^{-1/2})$. The expectation of the second term on the right hand side is $M(\gamma_{0})$. Inserting the expansion from (2.7) into the third term one obtains

[TABLE]

Applying a Hoeffding decomposition to the U-statistic term (C.15) one sees that the degenerate part is negligible and the dominating part is the (centred) linear term $n^{-1/2}\sum_{i=1}^{n}\delta(Z_{i})$ with $\delta$ from Theorem 3.4. The assertion of the theorem now follows from the classical central limit theorem. $\Box$

**Supplement to:

“Specification testing in semi-parametric transformation models” by Nick Kloodt, Natalie Neumeyer and Ingrid Van Keilegom**

Appendix D Bootstrap theory

In this section, we use the notations for the probability space as in section 4. The expectation with respect to $P_{2}^{1}(\omega,\cdot)$ is written as $E[\cdot|\omega]$. Note that the functions $h^{*}$ and $\psi^{*}$ depend on $\omega$ via the original sample. This is suppressed in the notation. We formulate the following additional assumptions.

1. (A8enumi)

The following properties are meant conditional on the data $(Y_{i},X_{i}),i=1,...,n,$ and thus define for fixed $n\in\mathbb{N}$ some subsets $A_{n}\in\mathcal{A}_{1}$ of $\Omega_{1}$, where these properties are valid. Thus, let $\omega\in A_{n}$, then we assume the following.

1. (i)

The transformation estimator $\hat{h}^{*}$ fulfils

[TABLE]

for all $\delta>0$, for some function $\psi^{*}$ and $h^{*}$ from Algorithm 4.1. 2. (ii)

For $\mathcal{U}_{0}$ from (A8) and $\mathcal{Y}_{w}$ from (A2) we have $\mathcal{T}_{S^{*}}(h^{*}(\mathcal{Y}_{w}))\subset\mathcal{U}_{0}$. 3. (iii)

Let $Z^{*}=(U^{*},X^{*})=(\mathcal{T}^{*}(Y^{*}),X^{*})$. The function class $\{z\mapsto\psi^{*}(z,t):t\in\mathcal{U}_{0}\}$ is Donsker (for fixed $n$, but $m\rightarrow\infty$) with respect to $P_{2}^{1}(\omega,\cdot)^{Z^{*}}$ (distribution of $Z^{*}$ conditional on $\omega$) and

[TABLE] 4. (iv)

The fourth moment $E[w(h_{0}^{-1}(S_{1}^{*}))\psi^{*}(Z_{1}^{*},U_{1}^{*})^{4}|\omega]$ is finite and the conditional moments $E[w({h^{*}}^{-1}(S_{1}^{*}))\psi^{*}(Z_{1}^{*},U_{2}^{*})^{2}|Z_{1}^{*}=z,\omega]$ are locally bounded. 5. (v)

For all compact sets $\mathcal{K}\subseteq\mathbb{R}$ we have

[TABLE] 6. (vi)

One has $\underset{y\rightarrow z}{\lim}\,E\left[|\psi^{*}(z,U_{1}^{*})-\psi^{*}(y,U_{1}^{*})|\,|\omega\right]=0$ for all $z$ in the support of $Z^{*}$.

For $A_{n}$ as defined above, we assume $P(A_{n})\rightarrow 1$ for $n\rightarrow\infty$.

1. (A9enumi)

Define the distribution function of $Z^{*}$ for some $\omega\in\Omega_{1}$ by $F_{Z^{*}}(z)=P_{2}^{1}(\omega,\{Z^{*}\leq z\})$ and assume

[TABLE]

Moreover, for all compact $\mathcal{K}\subseteq\mathbb{R}^{d_{X}+1}$ there exists an appropriate $C>0$, such that for $n\to\infty$

[TABLE]

Further,

[TABLE]

for all $z\in\mathbb{R}^{d_{X}+1}$, $s\in\mathbb{R}$ and for $\psi$ from (A8) for $n\to\infty$.

D.1 Main bootstrap results and proofs

Theorem D.1**.**

Assume $H_{0}$,(A1)–(A8),(A8enumi),(A9enumi). Then, the bootstrap statistic $T_{n,m}^{*}$ computed by Algorithm 4.1 fulfils (4.2). If $q_{\alpha}^{*}$ denotes for all $\alpha\in(0,1)$ the corresponding bootstrap quantile described in Algorithm 4.1, it is

[TABLE]

for all $\delta>0$.

Proof of Theorem D.1. As in the proof of Lemma D.5, the conditional distribution and expectation of $(Y_{1}^{*},X_{1}^{*}),...,(Y_{m}^{*},X_{m}^{*})$ given $(Y_{1},X_{1}),...,(Y_{n},X_{n})$ are denoted by $P^{*}$ and $E^{*}$, respectively. Consider $\omega\in A_{n}$ with $A_{n}$ from (A8enumi). The proof can be divided into two parts: First, the uniform convergence of some bootstrap components appearing in the asymptotic distribution of the bootstrap test statistic is proven and second, the assertion itself is shown by the convergence of the conditional distribution functions in probability. Referring to the definition of $h^{*}$, the following condition (A6enumi) is valid.

1. (A6enumi)

With probability converging to one, minimizing the function $M^{*}:\Upsilon\rightarrow\mathbb{R},$

[TABLE]

leads to a unique solution

[TABLE]

in the interior of $\Upsilon$.

Here, uniqueness follows due to $E^{*}[w(Y^{*})(h^{*}(Y^{*})(\Lambda_{\hat{\theta}}(1)-\Lambda_{\hat{\theta}}(0))+\Lambda_{\hat{\theta}}(0)-\Lambda_{\hat{\theta}}(Y^{*}))^{2}]=0$ from (A6). With the notations

[TABLE]

a function $\zeta^{*}$ can be defined as

[TABLE]

Moreover, define

[TABLE]

and $b^{*}=E[\zeta^{*}(Z_{1}^{*},Z_{1}^{*})|\omega]$, where $W_{1},W_{2},...$ are independent and standard normally distributed and $\lambda_{1}^{*},\lambda_{2}^{*},...$ are the eigenvalues of the operator

[TABLE]

Therefore for fixed $n$ and conditional on $(X_{1},Y_{1}),\dots,(X_{n},Y_{n})$, one can proceed exactly as in the proof of Theorem 3.1 to obtain that $T_{n,m}^{*}$ converges in distribution to $T_{n}^{*}$ for $m\rightarrow\infty$. We have for $n\to\infty$

[TABLE]

as well as

[TABLE]

Convergence of the bootstrap components: In the following, the convergence in probability of $h^{*},R^{*},\Gamma^{*},\varphi^{*},\zeta^{*},b^{*}$ to $h_{0}$ (the true transformation under $H_{0}$), $R(s)$ from (3.3), $\Gamma_{0}$ from (3.4), $\varphi$ from (3.5), $\zeta$ from (3.6) and $b$ from (C.9) is shown.

One has $\hat{\theta}=\theta_{0}+o_{P}(1)$ for $n\to\infty$. From (4.1) follows $h^{*}=h+o_{P}(1)$ for $n\to\infty$ uniformly on compact sets and thus $w((h^{*})^{-1}(s))R^{*}(s)=w(h^{-1}(s))R(s)+o_{P}(1)$ uniformly in $s\in\mathbb{R}$. Further, there exists some $\tilde{C}>0$, such that $h^{*}$ is bijective on $\mathcal{Y}_{w}$ and $|h^{*}|<\tilde{C}$ as well as $|(h^{*})^{-1}|<\tilde{C}$ on $\mathcal{Y}_{w}$ or $h^{*}(\mathcal{Y}_{w})$ with probability converging to one, that is

[TABLE]

for $n\to\infty$. This in turn means that (see Remark 4.2 for a possible adjustment of $h^{*}$)

[TABLE]

for $n\to\infty$ for some sufficiently large $C>0$. Let $f_{S},f_{S^{*}}$ denote the densities of $S$ and $S^{*}$, respectively, conditioned on $(Y_{i},X_{i}),i=1,...,n$. The dominated convergence theorem leads to (the inequality is meant componentwise)

[TABLE]

for all $\delta>0$. Consequently,

[TABLE]

for $n\rightarrow\infty$. Due to part (D.2) of (A9enumi), the map $(Z_{1}^{*},U_{2}^{*})\mapsto w((h^{*})^{-1}(S_{2}^{*}))\psi^{*}(Z_{1}^{*},U_{2}^{*})$ is bounded by some constant $C$ uniformly over compact sets with probability converging to one. Together with the dominated convergence theorem, this leads to boundedness of $\varphi^{*}$ as well as $\varphi^{*}(z)=\varphi(z)+o_{P}(1)$ and finally boundedness of $\zeta^{*}$ and

[TABLE]

for all $z_{1},z_{2}\in\mathbb{R}^{d_{X}+1}$. Additionally, one has

[TABLE]

for some $C>0$, so that for all $\delta>0$

[TABLE]

that is $b^{*}=b+o_{P_{1}}(1)$ for $n\to\infty$. Now, all ingredients to prove the convergence of the distribution functions $F_{T_{n}^{*}}(t)=P^{*}(T_{n}^{*}\leq t)$ to $P(T\leq t)$ in probability have been presented.

Convergence of the distribution functions: Let $t\in\mathbb{R},\varepsilon>0$ be arbitrary and $\tilde{\varepsilon},\delta_{\tilde{\varepsilon}}>0,M_{\tilde{\varepsilon}}\in\mathbb{N}$ such that

[TABLE]

[TABLE]

(due to the proof of Theorem 3.1 for sample size $n$ replaced by $m$) and

[TABLE]

For a moment consider $m$ as fixed and define

[TABLE]

Then, $P((Z_{1}^{*},Z_{2}^{*})\in\mathcal{K}_{\tilde{\varepsilon},m})\overset{n\rightarrow\infty}{\longrightarrow}1$. For all $m\geq M_{\varepsilon}$ one has

[TABLE]

Here, the equality follows from the Portmanteau theorem due to (D.1). The same reasoning leads to

[TABLE]

for all $\varepsilon>0$ and thus

[TABLE]

Let $M\in\mathbb{N}$ and define

[TABLE]

as well as

[TABLE]

Note that $P_{1}(B_{n})\overset{n\rightarrow\infty}{\longrightarrow}1$ because of (D.5) and $P_{1}(B_{n}\cap A_{M,\varepsilon})\overset{M\rightarrow\infty}{\longrightarrow}P_{1}(B_{n})$. Let $N\in\mathbb{N}$ such that $P_{1}(B_{N})\geq 1-\varepsilon$ and let $M_{\varepsilon}$ fulfil (D.6) and $P_{1}(B_{N})\leq P_{1}(B_{N}\cap A_{M_{\varepsilon},\varepsilon})+\varepsilon$. Then, it is

[TABLE]

Since $\varepsilon>0$ can be chosen arbitrarily small, one has

[TABLE]

In total, (4.2) was proven, that is,

[TABLE]

It remains to deduce

[TABLE]

from this. Let $\delta>0$ be arbitrarily small and let $q_{\alpha}$ be the $\alpha$-quantile of $T$ and define

[TABLE]

which depends on $n$, but converges to a positive value. Then, if

[TABLE]

one has

[TABLE]

for $n$ sufficiently large, and thus $q_{\alpha}^{*}>q_{\alpha}-\delta$. Analogously, \big{|}P_{2}^{1}(\omega,\{T_{n,m}^{*}\leq q_{\alpha}+\delta\})-P(T_{n}\leq q_{\alpha}+\delta)\big{|}\leq\varepsilon implies $P_{2}^{1}(\omega,\{T_{n,m}^{*}\leq q_{\alpha}+\delta\})>\alpha$ and $q_{\alpha}^{*}<q_{\alpha}+\delta$, so that in total

[TABLE]

$\Box$

Theorem D.2**.**

Assume $H_{1}$,(A1)–(A4),(A6’),(A8enumi). Then, the bootstrap statistic $T_{n,m}^{*}$ computed by Algorithm 4.1 fulfils (4.3). If $q_{\alpha}^{*}$ denotes for all $\alpha\in(0,1)$ the corresponding bootstrap quantile described in Algorithm 4.1, it is

[TABLE]

Proof of Theorem D.2. Borrowing the notations for $R^{*},\Gamma^{*}$ and $\varphi^{*}$ from the proof of Theorem D.1 remind that $\zeta^{*}$ can be written as

[TABLE]

Similar to the proof of Theorem D.1 Assumption (A6’) leads to (A6enumi) from the proof of Theorem D.1. As before, Equations (D.4) and (D.5) remain valid. Especially, it is

[TABLE]

for $b^{*}=E[\zeta^{*}(Z_{1}^{*},Z_{1}^{*})|\omega]$. Since (A8enumi) ensures that

[TABLE]

and all compact sets $\mathcal{K}\subseteq\mathbb{R}$, it is

[TABLE]

(boundedness of $R^{*}$ and $\Gamma^{*}$ follows as in the proof of Theorem D.1). The same reasoning as in the proof of Theorem 3.1 leads to

[TABLE]

so that

[TABLE]

and thus $\underset{m\rightarrow\infty}{\limsup}|T_{n}^{*}|=o_{P}(n)$. Referring to Theorem 3.4, one has

[TABLE]

with $M(\gamma_{0})>0$. Consequently,

[TABLE]

In total, (4.3) was proven, that is,

[TABLE]

It remains to deduce

[TABLE]

for all $\alpha\in(0,1)$ from this. Let $\alpha\in(0,1)$ be fixed. If $T_{n}>q_{\alpha}^{*}$, one has

[TABLE]

so that

[TABLE]

$\Box$

D.2 Nonparametric transformation estimation in the bootstrap case

The aim of this subsection is to show that the estimating approach developed by Colling and Van Keilegom (2019) can be applied in this context. To this end, for the estimator $\hat{h}$ from Appendix B respectively its bootstrap analog $\hat{h}^{*}$ validity of assumptions (A8enumi) and (A9enumi) needs to be shown, such that Theorems D.1 and D.2 apply and Algorithm 4.1 gives valid approximation of the critical value. Denote the conditional density of $\varepsilon({\theta_{0}})$ (defined in Algorithm 4.1) given $X$ by $f_{\varepsilon({\theta_{0}})|X}$. Then we need the following assumptions.

1. (A10)

Denote the conditional density of $\varepsilon({\theta_{0}})$ as defined in Algorithm 4.1 given $X$ by $f_{\varepsilon({\theta_{0}})|X}$. Let $\mathcal{K}\subseteq\mathbb{R}$ be compact and $\ell$ be bounded and $r$-times continuously differentiable with bounded derivatives and denote the $k$-th derivative of $\ell$ by $\ell^{(k)}$. Further, assume

[TABLE]

for $l\in\{0,j\}$ and

[TABLE]

as well as

[TABLE]

for sufficiently large $C>0,n\in\mathbb{N}$ and all $j=1,...,r-1$. Moreover, let

[TABLE]

and

[TABLE]

1. (B11enumi)

In the following, the notations from Algorithm 4.1 are employed. Let $1/(na_{n})\to 0$, $(\log n)/(nb_{n}^{d_{X}+4})\to 0$.

1. (1)

Let $\hat{g}$ be $(q+2)$-times continuously differentiable (same $q$ as in (B4)). 2. (2)

Assume

1. $\bullet$

$\kappa$ and $\ell$ are $(q+2)$-times continuously differentiable, 2. $\bullet$

$\kappa$ has bounded support, 3. $\bullet$

$\kappa(0)>0$ and $\ell>0$, 4. $\bullet$

it is either \big{|}\frac{\partial}{\partial u}\ell(u)\big{|}\leq K,

[TABLE]

for all $|u|>L$ and \tilde{f}\in\Big{\{}\ell,\frac{\partial}{\partial u}\ell\Big{\}} or

[TABLE]

for some $\nu>1,K,L\in(0,\infty)$ and all \tilde{f}\in\Big{\{}\kappa,\frac{\partial}{\partial x_{i}}\kappa,\frac{\partial^{2}}{\partial x_{i}^{2}}\kappa\Big{\}}, where the same $i$ was used as in (B3). From now on, the case $i=1$ is considered w.l.o.g.

Lemma D.3**.**

Let $\mathcal{K}\subset\mathbb{R}$ be compact. Assume (A10) and

[TABLE]

Then,

[TABLE]

and

[TABLE]

Remark D.4**.**

1. (i)

Later, it will be shown that $\frac{1}{n}\sum_{l=1}^{n}\hat{\varepsilon}_{l}=o_{P}(1)$ in the proof of Lemma D.6. Hence, the Assertion of Lemma D.3 can be extended to the centred residuals $\tilde{\varepsilon}_{i}$ by considering a slightly larger set $\mathcal{K}$. 2. (ii)

Many of the assumptions in (A10) can be replaced by less complex, but more restrictive versions. For example, due to (D.14) assumption (D.10) is implied by

[TABLE]

or

[TABLE]

for all $j=1,...,r-1.$ 3. (iii)

It is ||\hat{\theta}-\theta_{0}||=O_{P}\big{(}n^{-\frac{1}{4}}\big{)} under the alternative. Unfortunately, the experimenter in advance does not know, if the null hypothesis or the alternative holds, so that in general (D.14) limits $a_{n}$ to a_{n}^{-1}=o\big{(}n^{\frac{r}{4(r+1)}}\big{)}=o\big{(}n^{\frac{1}{4}}\big{)}. 4. (iv)

$g_{\theta_{0}}$* can be estimated using the Nadaraya-Watson approach as in Heuchenne et al. (2015). Under some additional assumptions, their Proposition 6.1 or to be precise its extension in the supplementary material of Colling and Van Keilegom (2016, p. 7) yields $\hat{g}(x)-g_{\theta_{0}}(x)=O_{P}(((\log n)/(nh_{x}^{d_{X}}))^{1/2}+||\hat{\theta}-\theta_{0}||)$ uniformly on compact sets. When assuming the existence of some compact set $\mathcal{K}\subseteq\operatorname{supp}(f_{X})$ such that $\operatorname{supp}(v)$ is contained in the interior of $\mathcal{K}$, Equation (D.14) (and a counterpart of (D.17) below for compact sets) can be obtained when discarding those $(Y_{i},X_{i}),(Y_{j}^{*},X_{j}^{*})$ such that $X_{i},X_{j}^{*}\notin\mathcal{K}$ (note that an equation similar to (2.7) can still be derived (although in general with another $\psi$)). Then, (D.14) requires $a_{n}^{-1}=o(((\log n)/(nh_{x}^{d_{X}}))^{-\frac{r}{2(r+1)}}+n^{\frac{r}{4(r+1)}})$ for ||\hat{\theta}-\theta_{0}||=O_{P}\big{(}n^{-\frac{1}{4}}\big{)}.*

Proof of Lemma D.3. Only the second assertion is shown since the first one can be concluded similarly. The proof uses similar techniques as Han2008. First, for the deviation terms $R_{i}=\hat{\varepsilon}_{i}-\varepsilon_{i}(\theta_{0})$ and appropriate $u_{i}^{*},i=1,...,n,$ a Taylor expansion leads to

[TABLE]

For appropriate $\tilde{\theta}_{y}$ between $\hat{\theta}$ and $\theta_{0}$ the $R_{i}$ can be split into

[TABLE]

Therefore,

[TABLE]

for all $j=1,...,r-1$ and

[TABLE]

for some sufficiently large constant $C>0$, so that it suffices to treat the cases $R_{i}^{(1)}=\hat{g}(X_{i})-g_{\theta_{0}}(X_{i})$ and $R_{i}^{(2)}=\tilde{R}_{i}$ separately.

When inserting $R_{i}^{(1)}$ in equation (D.15) negligibility of the last summand directly follows from (D.14) and the boundedness of $\ell^{(r)}$. Thanks to Han2008, to prove

[TABLE]

for all $j=0,...,r-1$ and some constant $C>0$, it suffices to show uniform (with respect to $u$) boundedness of the expectation. Hence one has

[TABLE]

for some constant $C>0$ (see (D.8)) and thus

[TABLE]

for all $j=1,...,r-1$. Further $\tilde{R}_{i}$ can be written as

[TABLE]

where the $O_{P}$-terms are independent of $i$. When inserting $\tilde{R}_{i}$ in equation (D.15), one has for any $\delta>0$

[TABLE]

for all $j=1,...,r-1$. By assumptions (D.8) and (D.9) the expected value of the sum

[TABLE]

can be bounded by some constant $C>0$, so that

[TABLE]

by (D.10) and (D.14). The remaining term can be treated similarly by applying (D.11) and (D.12) to obtain

[TABLE]

Altogether one obtains

[TABLE]

uniformly on compact sets. $\Box$

Lemma D.5**.**

Let Assumptions (B11enumi) and (D.14) be fulfilled. Further, assume (A1)–(A8),(A10),(B1)–(B10),

[TABLE]

Further, assume the existence of a neighbourhood $\tilde{\Theta}$ of $\theta_{0}$ such that the map $y\mapsto\Lambda_{\theta}(y)$ is $(q+2)$-times continuously differentiable for all $\theta\in\tilde{\Theta}$. Let $\hat{h}^{*}$ be the estimator from (B.4) based on the bootstrap data $(Y_{j}^{*},X_{j}^{*}),j=1,...,m$. Assume that the density of $\varepsilon(\theta_{0})$ is continuous and

[TABLE]

Then, assumptions (A8enumi) and (A9enumi) are fulfilled.

Proof of Lemma D.5. Note that conditional on $(Y_{1},X_{1}),...,(Y_{n},X_{n})$ the random variables $(Y_{1}^{*},X_{1}^{*}),...,(Y_{m}^{*},X_{m}^{*})$ are independent as well as identically distributed. Moreover, after conditioning on the original data, the Assumptions (B1)–(B10) are valid for the bootstrap sample with probability converging to one, so that due to Remark 4.2 the same reasoning as in (Colling and Van Keilegom, 2019) can be applied to obtain (4.4).

For notational convenience the conditional distribution of $(Y_{1}^{*},X_{1}^{*}),...,(Y_{m}^{*},X_{m}^{*})$ conditional on $(Y_{1},X_{1}),...,(Y_{n},X_{n})$ is written as $P^{*}$ and the expectation with respect to $P^{*}$ is written as $E^{*}$. Let $F_{Y^{*}|X^{*}}$ denote the conditional distribution function of $Y_{1}^{*}$ conditioned on $X_{1}^{*}$ (and $(Y_{1},X_{1}),...,(Y_{n},Y_{n})$). To verify (A8enumi) $\psi^{*}$ has to be examined further and to define $\psi^{*}$ some further notations are needed. Let $v$ be the weighting function from assumption (B7) and define

[TABLE]

and (for $\tilde{v}^{*}=\tilde{v}_{1}^{*},\tilde{v}_{2}^{*}$)

[TABLE]

where $D_{p,0}^{*}(u,x),...,D_{f,1}^{*}(u,x)$ are defined as

[TABLE]

with $\Phi^{*}(u|x)=F_{U^{*}|X^{*}}(u,x)=\frac{p^{*}(u,x)}{f_{X^{*}}(x)},\Phi^{*}_{u}(u|x)=\frac{\partial}{\partial_{u}}\Phi^{*}(u,x)$, $\Phi^{*}_{i}(u|x)=\frac{\partial}{\partial x_{1}}\Phi^{*}(u,x)$ and $f_{X^{*}}$ from Algorithm 4.1. Then, $\psi^{*}$ is defined as

[TABLE]

Condition (A8enumi) for $\psi^{*}$ is implied by the same reasoning as in Colling and Van Keilegom (2019). Note that the first part of Remark 4.2 ensures that $v$ can be used as the weighting function for the bootstrap data as well.

To prove (A9enumi), an auxiliary lemma is shown in the following. Thanks to the expressions above for $\psi^{*},D_{p,0}^{*}(u,x),...,D_{f,1}^{*}(u,x)$, equations (D.1) and (D.3) will be a direct consequence of Lemma D.6, while (D.2) follows from expression (D.21), so that $\psi^{*}$ fulfils (A9enumi) then.

Lemma D.6**.**

Let \mathcal{C}\subseteq\big{(}-\frac{F_{Y}(0)}{F_{Y}(1)-F_{Y}(0)},\frac{1-F_{Y}(0)}{F_{Y}(1)-F_{Y}(0)}\big{)} be a compact set and define

[TABLE]

Under the assumptions of Lemma D.5, one has

[TABLE]

Here, $p$ is defined as in B.5 and $f_{X^{*}}$ is defined as in Algorithm 4.1.

Proof: Most of the proof contains in applying the results in Han2008 for kernel estimates. While doing so, note that due to (D.13) and (D.18) kernel estimates like

[TABLE]

converge uniformly in $e\in\mathbb{R}$ to their expectation (see Theorem 4 in Han2008).

The results for $f_{X^{*}}$ directly follow from Han2008 (note that $\kappa$ is a kernel of order $2$). The assertion for $\frac{\partial}{\partial x_{1}}f_{X^{*}}$ and $\frac{\partial^{2}}{\partial x_{1}^{2}}f_{X^{*}}$ follows similarly by applying for example E\big{[}\frac{\partial^{2}}{\partial v_{i}^{2}}f_{X^{*}}(v)\big{]}=\frac{\partial^{2}}{\partial x_{1}^{2}}f_{X}(x)+\mathcal{O}(b_{n}^{2}). Moreover, $p^{*}$ can be expressed for any $j\in\{1,...,m\}$ as

[TABLE]

where $(-\infty,x]=\times_{i=1}^{d_{X}}(-\infty,x_{i}]$ and $F_{\xi}$ denotes the cumulative distribution function corresponding to $\ell$. From now on, only $\frac{\partial}{\partial x_{1}}p^{*}$ is considered, since the other terms can be treated analogously. Due to (D.16), one obtains (if $||\hat{\theta}-\theta_{0}||\leq\delta$)

[TABLE]

Since $P(||\hat{\theta}-\theta_{0}||\leq\delta)\rightarrow 1$, this can be used together with Lemma D.3 to obtain

[TABLE]

uniformly with respect to $x\in\operatorname{supp}(v)$ and with respect to $u$ belonging to some compact set $\mathcal{K}$, where the third last equality again follows from Theorem 4 in Han2008. The same reasoning for $\frac{\partial}{\partial u}p^{*}$ results in

[TABLE]

uniformly in $(u,x)\in\mathcal{K}\times\operatorname{supp}(v)$. Similarly, one can show

[TABLE]

as well as

[TABLE]

uniformly on compact sets. Hence, after possibly adjusting the set of admissible values for $u$, (D.22) leads to

[TABLE]

uniformly on $(u,x)\in\mathcal{C}\times\operatorname{supp}(v)$. $\Box$

Remark D.7**.**

Roughly speaking, the proof of Lemma D.5 was based on the convergence of $\psi^{*}$ to $\psi$. If the alternative holds, it is not even clear if $\psi^{*}$ stabilizes in some sense (see Assumption (A8enumi)(v)). Hence, additional assumptions are needed. For that purpose define

[TABLE]

While doing so, assume $F_{S}^{B}(0)<F_{S}^{B}(1)$ to ensure that $\mathcal{T}_{S}^{B}$ is well defined, and define

[TABLE]

$\tilde{\Phi}$* plays a similar role under the alternative as $\Phi(u|x)=F_{U|X}(u|x)$ under the null hypothesis and thus needs to be continuously differentiable on $\mathcal{U}_{0}\times\operatorname{supp}(v)$ with (again, the same $i$ as in (B3) is used)*

[TABLE]

Lemma D.8**.**

Let Assumptions (B11enumi) and (D.14) be fulfilled. Further, assume $H_{1}$,(A1)–(A4),(A6’),(A7),(A8’),(A10), (B1)–(B10),(D.17), (D.18) and (D.23). Further, assume the existence of a neighbourhood $\tilde{\Theta}$ of $\theta_{0}$ such that the map $y\mapsto\Lambda_{\theta}(y)$ is $(q+2)$-times continuously differentiable for all $\theta\in\tilde{\Theta}$. Let $\hat{h}^{*}$ be the estimator from (B.4) based on the bootstrap data $(Y_{j}^{*},X_{j}^{*}),j=1,...,m$. Then, assumption (A8enumi) is fulfilled.

Proof of Lemma D.8. Only equation (A8enumi)(v) needs to be proven, since the remaining conditions follow as in the proof of Lemma D.5 by the results of Colling and Van Keilegom (2019). In contrast to the proof of Lemma D.5, it is \hat{\theta}-\theta_{0}=O_{P}\big{(}n^{-\frac{1}{4}}\big{)} (here and in the following, $o_{P}$- and $O_{P}$-terms are with respect to $P_{1}$ and for $n\rightarrow\infty$) and the asymptotic behaviour of $\psi^{*}$ can not be reduced to the convergence to $\psi$. Nevertheless, $\psi^{*}$ can be expressed as in (D.21) with $D_{p,0}^{*},...,D_{f,1}^{*}$ as in (D.19)–(D.20). The main idea is to prove uniform convergence of $\frac{\partial}{\partial u}\Phi^{*}$ and $\frac{\partial}{\partial u}\Phi^{*}$ on $\mathcal{U}_{0}\times\operatorname{supp}(v)$ to $\frac{\partial}{\partial u}\tilde{\Phi}$ and $\frac{\partial}{\partial x_{1}}\tilde{\Phi}$, respectively, while the remaining parts of ${\delta_{j}^{*}}^{\tilde{v}_{1}^{*}},{\delta_{j}^{*}}^{\tilde{v}_{2}^{*}}$ and ${Q^{*}}^{\prime}$ are bounded in probability.

Due to (D.23) it is \mathcal{U}_{0}\subseteq\big{(}-\frac{F_{S}^{B}(0)}{F_{S}^{B}(1)-F_{S}^{B}(0)},\frac{1-F_{S}^{B}(0)}{F_{S}^{B}(1)-F_{S}^{B}(0)}\big{)}. In the following it is proven that under the assumptions of Lemma D.8 one has

[TABLE]

Due to Lemma D.3 $F_{S^{*}}$ can be written for appropriate $u_{i,w}^{*}\in\mathbb{R},i=1,...,n,\ w\in\operatorname{supp}(\kappa),$ as

[TABLE]

As a distribution function $F_{\xi}$ is bounded so that

[TABLE]

that is

[TABLE]

Since $F_{S^{*}}$ and $F_{S}^{B}$ are distribution functions, this leads to the uniform convergence

[TABLE]

and thus to (D.24). To prove (D.25) write $F_{\varepsilon(\theta_{0})}$ as

[TABLE]

which implies

[TABLE]

for

[TABLE]

Since $h$ and $h_{\theta_{0}}$ are strictly increasing and $F_{S}^{B}$ is continuous, it is $F_{S}^{B}(u_{1})<F_{S}^{B}(u_{2})$ for all u_{1}<u_{2}\in(\mathcal{T}_{S}^{B})^{-1}(\mathcal{U}_{0})\subseteq(\mathcal{T}_{S}^{B})^{-1}\Big{(}\big{(}-\frac{F_{S}^{B}(0)}{F_{S}^{B}(1)-F_{S}^{B}(0)},\frac{1-F_{S}^{B}(0)}{F_{S}^{B}(1)-F_{S}^{B}(0)}\big{)}\Big{)}. Especially, $(\mathcal{T}_{S}^{B})^{-1}$ is strictly increasing on $\mathcal{U}_{0}$, that is, (D.25) follows from (D.24).

Finally, this can be used to obtain

[TABLE]

uniformly in $(u,x)\in\mathcal{U}_{0}\times\operatorname{supp}(v)$, where the second last equality follows from the continuity of $f_{\varepsilon(\theta_{0})}$. The bootstrap functions

[TABLE]

$\Phi^{*}$ and $\frac{\partial}{\partial u}\Phi^{*}$ can be treated by similar arguments to obtain

[TABLE]

Since $Q^{*}=\mathcal{T}_{S^{*}}^{-1}$ equations (D.25) and (D.19)–(D.20) lead to

[TABLE]

$\Box$