Limit distribution theory for block estimators in multiple isotonic regression

TL;DR

This paper derives the limit distribution for a block estimator in multiple isotonic regression, revealing how the estimator's distribution depends on the local smoothness of the regression function and the design, extending known results to multivariate settings.

Contribution

It provides the first limit distribution theory for the max-min block estimator in multivariate isotonic regression, including explicit forms and optimality results.

Findings

Limit distribution characterized by effective dimensions and sample size.

Distribution generalizes the Chernoff distribution to multivariate cases.

Results are optimal in local asymptotic minimax sense.

Abstract

We study limit distributions for the tuning-free max-min block estimator originally proposed in [FLN17] in the problem of multiple isotonic regression, under both fixed lattice design and random design settings. We show that, if the regression function $f_0$ admits vanishing derivatives up to order $\alpha_k$ along the $k$-th dimension ($k=1,\ldots,d$) at a fixed point $x_0 \in (0,1)^d$, and the errors have variance $\sigma^2$, then the max-min block estimator $\hat{f}_n$ satisfies \begin{align*} (n_\ast/\sigma^2)^{\frac{1}{2+\sum_{k \in \mathcal{D}_\ast} \alpha_k^{-1}}}\big(\hat{f}_n(x_0)-f_0(x_0)\big)\rightsquigarrow \mathbb{C}(f_0,x_0). \end{align*} Here $\mathcal{D}_\ast, n_\ast$, depending on $\{\alpha_k\}$ and the design points, are the set of all `effective dimensions' and the size of `effective samples' that drive the asymptotic limiting distribution, respectively. If furthermore either $\{\alpha_k\}$ are relative primes to each other or all mixed derivatives of $f_0$ of certain critical order vanish at $x_0$, then the limiting distribution can be represented as $\mathbb{C}(f_0,x_0) =_d K(f_0,x_0) \cdot \mathbb{D}_{\alpha}$, where $K(f_0,x_0)$ is a constant depending on the local structure of the regression function $f_0$ at $x_0$, and $\mathbb{D}_{\alpha}$ is a non-standard limiting distribution generalizing the well-known Chernoff distribution in univariate problems. The above limit theorem is also shown to be optimal both in terms of the local rate of convergence and the dependence on the unknown regression function whenever such dependence is explicit (i.e. $K(f_0,x_0)$), for the full range of $\{\alpha_k\}$ in a local asymptotic minimax sense.

Equations

$$\displaystyle(n_{\ast}/\sigma^{2})^{\frac{1}{2+\sum_{k\in\mathcal{D}_{\ast}}\alpha_{k}^{-1}}}\big{(}\hat{f}_{n}(x_{0})-f_{0}(x_{0})\big{)}\rightsquigarrow\mathbb{C}(f_{0},x_{0}).$$

$$\displaystyle n^{1/3}\big{(}\hat{f}_{n}(x_{0})-f_{0}(x_{0})\big{)}\rightsquigarrow\lvert f^{\prime}_{0}(x_{0})f_{0}(x_{0})/2\rvert^{1/3}\mathbb{Z}.$$

$$\displaystyle n^{2/5}\big{(}\hat{f}_{n}(x_{0})-f_{0}(x_{0})\big{)}\rightsquigarrow\big{(}f_{0}^{\prime\prime}(x_{0})f_{0}^{2}(x_{0})/24\big{)}^{1/5}\mathbb{H}^{\prime\prime}(0),$$

$$Y_i=f_0(X_i)+{\xi }_i,i=1,\dots ,n,$$

$${\mathcal{F}}_d\equiv \{f:[0,1{]}^d\to \mathbb{R},f(x)\le f(y)\text{ if }{x}_i\le y_i\text{ for all }i=1,\dots ,d\}.$$

$${\hat{f}}_n(x_0)$$

$$=\underset{x_u\le x_0}{\max }\underset{\begin{array}{c}{x}_v\ge x_0\\ [x_u,x_v]\cap \{X_i\}\ne \varnothing \end{array}}{\min }\bar{Y}{|}_{[x_u,x_v]}$$

$$=\underset{x_u\le x_0}{\max }\underset{\begin{array}{c}{x}_v\ge x_0\\ [x_u,x_v]\cap \{X_i\}\ne \varnothing \end{array}}{\min }\bar{Y}{|}_{[x_u,1]\cap [0,x_v]},$$

$$\displaystyle(n_{\ast}/\sigma^{2})^{\frac{1}{2+\sum_{k\in\mathcal{D}_{\ast}}\alpha_{k}^{-1}}}\big{(}\hat{f}_{n}(x_{0})-f_{0}(x_{0})\big{)}\rightsquigarrow\mathbb{C}(f_{0},x_{0}).$$

$$\mathbb{C}(f_0,x_0){=}_{d}K(f_0,x_0)\cdot {\mathbb{D}}_{\mathbf{\alpha }}.$$

$$\displaystyle(n/\sigma^{2})^{\frac{1}{2+s}}\big{(}\hat{f}_{n}(x_{0})-f_{0}(x_{0})\big{)}\rightsquigarrow\bigg{\{}\prod_{k=1}^{s}\big{(}\partial_{k}f_{0}(x_{0})/2\big{)}\bigg{\}}^{\frac{1}{2+s}}\cdot\mathbb{D}_{(\underbrace{1,\ldots,1}_{s\textrm{ many }1\textrm{'s}},\infty,\ldots,\infty)}.$$

$$\bar{Y}{|}_A\equiv \frac{1}{n_A}\sum _{i:X_i\in A}{Y}_i,\overline{f_0}{|}_A\equiv \frac{1}{n_A}\sum _{i:X_i\in A}{f}_0(X_i),\bar{\xi }{|}_A\equiv \frac{1}{n_A}\sum _{i:X_i\in A}{\xi }_i$$

$$\displaystyle\lim_{\omega_{n}\searrow 0}\omega_{n}^{-1}\sup_{\begin{subarray}{c}x\in[0,1]^{d},\\ \lvert(x-x_{0})_{k}\rvert\leq L_{0}\cdot(r_{n})_{k},\\ 1\leq k\leq d\end{subarray}}\bigg{\lvert}f_{0}(x)-\sum_{\bm{j}\in J_{0}}\frac{\partial^{\bm{j}}f_{0}(x_{0})}{\bm{j}!}(x-x_{0})^{\bm{j}}\bigg{\rvert}=0.$$

$$\underset{0<{\omega }_n\le {\epsilon }_1}{\sup }\underset{\begin{array}{c}x\in [0,1{]}^d,\\ |(x-x_0{)}_k|\le (r_n{)}_k,\\ 1\le k\le d\end{array}}{\sup }{\omega }_n^{-1}|f_0(x)-f_0(x_0)|\le L_1;$$

$$0\le {\alpha }_1{\beta }_1\le \dots \le {\alpha }_s{\beta }_s\le \dots \le {\alpha }_d{\beta }_d\le \infty .$$

$${\mathcal{I}}_{\pi }^{[s+1:d]}(h_1,h_2)$$

$$\displaystyle\equiv\int_{\begin{subarray}{c}(x_{0}-h_{1})_{k}\leq x_{k}\leq(x_{0}+h_{2})_{k}\\ s+1\leq k\leq d\end{subarray}}\pi\big{(}(x_{0})_{1},\ldots,(x_{0})_{s},x_{s+1},\ldots,x_{d}\big{)}\ \mathrm{d}x_{s+1}\cdots\mathrm{d}x_{d},$$

$$\mathbb{C}(f_0,x_0)\equiv \underset{\begin{array}{c}{h}_1>0,\\ (h_1{)}_k\le (x_0{)}_k,\\ s+1\le k\le d\end{array}}{\sup }\underset{\begin{array}{c}{h}_2>0,\\ (h_2{)}_k\le (1-x_0{)}_k,\\ s+1\le k\le d\end{array}}{\inf }$$

$$\displaystyle\qquad\qquad\bigg{[}\frac{\mathbb{G}(h_{1},h_{2})}{\prod_{k=\kappa_{\ast}}^{s}\big{(}(h_{1})_{k}+(h_{2})_{k}\big{)}\mathcal{I}^{[s+1:d]}_{\pi}(h_{1},h_{2})}+\bar{f_{0}}(h_{1},h_{2};x_{0})\bigg{]},$$

$$\displaystyle\mathrm{Cov}\big{(}\mathbb{G}(h_{1},h_{2}),\mathbb{G}(h_{1}^{\prime},h_{2}^{\prime})\big{)}$$

$$\displaystyle=\prod_{k=\kappa_{\ast}}^{s}\big{(}(h_{1})_{k}\wedge(h_{1}^{\prime})_{k}+(h_{2})_{k}\wedge(h_{2}^{\prime})_{k}\big{)}\cdot\mathcal{I}^{[s+1:d]}_{\pi}\big{(}h_{1}\wedge h_{1}^{\prime},h_{2}\wedge h_{2}^{\prime}\big{)},$$

$$\overline{f_0}(h_1,h_2;x_0)\equiv \sum _{\begin{array}{c}\mathbf{j}\in J_{\ast },\\ j_k=0,1\le k\le {\kappa }_{\ast }-1\end{array}}\frac{{\partial }^{\mathbf{j}}{f}_0(x_0)}{(\mathbf{j}+1)!}\prod _{k={\kappa }_{\ast }}^s\frac{(h_2{)}_k^{j_k+1}-(-h_1{)}_k^{j_k+1}}{(h_2{)}_k+(h_1{)}_k}.$$

$${\mathbb{D}}_{\mathbf{\alpha }}$$

$$\displaystyle\qquad\qquad+\sum_{k=\kappa_{\ast}}^{s}\frac{(h_{2})_{k}^{\alpha_{k}+1}-(h_{1})_{k}^{\alpha_{k}+1}}{(h_{2})_{k}+(h_{1})_{k}}\bigg{]}.$$

$$\displaystyle(n_{\ast}/\sigma^{2})^{\frac{1}{2+\sum_{k=\kappa_{\ast}}^{s}\alpha_{k}^{-1}}}\big{(}\hat{f}_{n}(x_{0})-f_{0}(x_{0})\big{)}=\mathcal{O}_{\mathbf{P}}(1).$$

$$\displaystyle(n_{\ast}/\sigma^{2})^{\frac{1}{2+\sum_{k=\kappa_{\ast}}^{s}\alpha_{k}^{-1}}}\big{(}\hat{f}_{n}(x_{0})-f_{0}(x_{0})\big{)}\rightsquigarrow\mathbb{C}(f_{0},x_{0}).$$

$$\mathbb{C}(f_0,x_0){=}_{d}K(f_0,x_0)\cdot {\mathbb{D}}_{\mathbf{\alpha }},$$

$$\displaystyle\mathrm{Cov}\big{(}\mathbb{G}(h_{1},h_{2}),\mathbb{G}(h_{1}^{\prime},h_{2}^{\prime})\big{)}=\prod_{k=\kappa_{\ast}}^{d}\big{(}(h_{1})_{k}\wedge(h_{1}^{\prime})_{k}+(h_{2})_{k}\wedge(h_{2}^{\prime})_{k}\big{)},$$

$$\displaystyle\big{(}\pi(x_{0})n/\sigma^{2}\big{)}^{\frac{1}{2+\sum_{k=1}^{d}\alpha_{k}^{-1}}}\big{(}\hat{f}_{n}(x_{0})-f_{0}(x_{0})\big{)}\rightsquigarrow K(f_{0},x_{0})$$

$$\displaystyle\quad\times\sup_{h_{1}>0}\inf_{h_{2}>0}\bigg{[}\frac{\mathbb{G}(h_{1},h_{2})}{\prod_{k=1}^{d}\big{(}(h_{1})_{k}+(h_{2})_{k}\big{)}}+\prod_{k=1}^{d}\frac{(h_{2})_{k}^{\alpha_{k}+1}-(h_{1})_{k}^{\alpha_{k}+1}}{(h_{2})_{k}+(h_{1})_{k}}\bigg{]},$$

$${\omega }_n^{\mathrm{fixed}}$$

$${\omega }_n^{\mathrm{random}}$$

$$\displaystyle\omega=\frac{1}{\sqrt{\prod_{k=\ell}^{d}\big{(}\omega^{1/\alpha_{k}}n^{\beta_{k}}\big{)}}}.$$

$$\displaystyle(n/\sigma^{2})^{\frac{1}{2+s/\alpha}}\big{(}\hat{f}_{n}(x_{0})-f_{0}(x_{0})\big{)}\rightsquigarrow\mathbb{C}(f_{0},x_{0}),\quad\alpha>(d-s)/2;$$

Full text

[TABLE]

Here $\mathcal{D}_{\ast},n_{\ast}$, depending on $\{\alpha_{k}\}$ and the design points, are the set of all ‘effective dimensions’ and the size of ‘effective samples’ that drive the asymptotic limiting distribution, respectively. If furthermore either $\{\alpha_{k}\}$ are relative primes to each other or all mixed derivatives of $f_{0}$ of certain critical order vanish at $x_{0}$, then the limiting distribution can be represented as $\mathbb{C}(f_{0},x_{0})=_{d}K(f_{0},x_{0})\cdot\mathbb{D}_{\bm{\alpha}}$, where $K(f_{0},x_{0})$ is a constant depending on the local structure of the regression function $f_{0}$ at $x_{0}$, and $\mathbb{D}_{\bm{\alpha}}$ is a non-standard limiting distribution generalizing the well-known Chernoff distribution in univariate problems. The above limit theorem is also shown to be optimal both in terms of the local rate of convergence and the dependence on the unknown regression function whenever such dependence is explicit (i.e. $K(f_{0},x_{0})$), for the full range of $\{\alpha_{k}\}$ in a local asymptotic minimax sense.

There are two interesting features in our local theory. First, the max-min block estimator automatically adapts to the local smoothness and the intrinsic dimension of the isotonic regression function at the optimal rate. Second, the optimally adaptive local rates are in general not the same in fixed lattice and random designs. In fact, the local rate in the fixed lattice design case is no slower than that in the random design case, and can be much faster when the local smoothness levels of the isotonic regression function or the sizes of the lattice differ substantially along different dimensions.

The research of Q. Han is partially supported by DMS-1916221. The research of C.-H. Zhang is partially supported by DMS-1513378, DMS-1721495 and IIS-1741390.

1. Introduction

1.1. Overview

Limit distribution theory for shape-restricted estimators is of fundamental importance in the area of statistical inference under shape restrictions. There are two main types of limit distribution theories so far available in the literature.

One line starts from the seminal contribution of [PR69], who showed that the limiting distribution of the maximum likelihood estimator (MLE) of a decreasing density on $[0,\infty)$ (known as Grenander estimator) at a fixed point is given by the following: Suppose the true density $f_{0}$ is decreasing on $[0,\infty)$ and continuously differentiable around $x_{0}\in(0,\infty)$ with $f_{0}^{\prime}(x_{0})<0$. Then the MLE $\hat{f}_{n}$ satisfies

[TABLE]

Here $\mathbb{Z}$, known as the Chernoff distribution, is the slope at zero of the least concave majorant of the process $t\mapsto\mathbb{B}(t)-t^{2}$, where $\mathbb{B}$ is the standard Brownian motion starting at [math]. Later on, [Gro89] gives an exact analytic characterization of the limiting Chernoff distribution, whereas [Gro85] suggests the ‘switching relation’ that quickly becomes popular as a powerful proof technique in univariate problems with monotonicity shape restrictions. The limiting Chernoff distribution arises in a number of different problems with univariate monotonicity shape restrictions, e.g., (1) estimation of a regression function [Bru70, vE58, Wri81], (2) estimation of a monotone failure rate [HW95, HZ94, PR70], (3) estimation in interval censoring models [Gro96, GW92], etc. We refer the reader to the recent survey [DL18] for extensive references in this direction.

Another line of limit theorems for shape restricted estimators is pioneered by [GJW01a, GJW01b], who studied limit distribution for the MLE of a convex decreasing density on $[0,\infty)$ and the least squares estimator (LSE) of a convex regression function at a fixed point. In the density setting, if the true density $f_{0}$ is convex decreasing on $[0,\infty)$ and twice continuously differentiable in a neighborhood of $x_{0}$ with $f_{0}^{\prime\prime}(x_{0})>0$, [GJW01b] showed that the MLE $\hat{f}_{n}$ satisfies

[TABLE]

where $\mathbb{H}$ is a particular upper invelope of an integrated two-sided Brownian motion plus $t^{4}$, cf. [GJW01a]. The process $\mathbb{H}$ appears in several other problems involving univariate convexity shape restrictions, e.g. for the MLE of a log-concave density on $\mathbb{R}$, cf. [BRW09], for the MLE of a convex bathtub-shaped hazard function, cf. [JW09b], for the Rényi-divergence estimators for $s$-concave densities on $\mathbb{R}$, cf. [HW16], etc. A generalized version of $\mathbb{H}$ appears in [BW07] in the context of $k$-monotone density estimation.

Limit theorems of types (1.1)-(1.2) are not only interesting from a statistical point of view, but are of theoretical value in their own rights. Indeed, these limit theorems and the proof techniques used therein serve as fundamental building blocks for numerous further developments, including likelihood based inferential methods [Ban07, BW01, DW18, GJ15], bootstrap in non-standard problems [Kos08, SBW10], estimation and inference with dependence structures [AH06, AS11, BBS16], limit theory for global loss functions and functionals [Dur07, DKL12, GHL99, Jan14, KL05], limit distribution theory for shape-restricted estimators of discrete functions [BDK17, BJ16, BJRP13, JW09a], limit distribution theory for split points in decision trees [BM07, BY02], cube-root asymptotics in more general settings [AS11, KP90], just to name a few.

Despite the wealth of limit distribution theories for univariate shape-restricted problems, much less is known in multi-dimensional settings. The only exception we are aware of is the recent work [AP18], in which asymptotic distributions for isotonized estimators are derived in the settings of multi-dimensional discrete isotonic regression and probability mass function estimation. The goal of this paper is to study limit theorems for shape-restricted estimators, with a focus on the problem of multiple isotonic regression in a continuous setting.

Here is our setup. Consider the regression model

[TABLE]

where $X_{1},\ldots,X_{n}$ are design points which can be either fixed or random, and $\xi_{1},\ldots,\xi_{n}$ are random errors. By multiple isotonic regression we assume that the regression function $f_{0}\in\mathcal{F}_{d}$, where $\mathcal{F}_{d}$ denotes the class of coordinate-wise nondecreasing functions on $[0,1]^{d}$:

[TABLE]

In addition to the aforementioned importance of having a limit distribution theory for shape restricted estimators beyond univariate settings, there is one further consideration for such a theory, related to one distinct attractive feature of shape-constrained models: the MLE/LSE often exists and automatically adapts to certain structures of the underlying truth without the need of any tuning. This automatic adaptation property has attracted a lot of recent attention, mostly from a global perspective. Indeed, adaptation of shape constrained MLEs/LSEs to piecewise simple structures in global metrics is confirmed extensively in various univariate models, cf. [Bel18, CGS15, GS15, KGS18, Zha02]. Typically, these tuning-free estimators adapt to piecewise constant/linear signals in univariate models with monotonicity/convexity shape constraints. For the multiple isotonic regression model (1.3), global adaptation of the LSE to piecewise constant signals is proved for the bivariate case $d=2$ in [CGS18], and for the case of general dimensions in [HWCS18]. See also [Han19]. Despite these very positive adaptation results, there remain two important drawbacks for considering global adaptation of the natural LSE in the multiple isotonic regression model:

1. (D1)

The isotonic regression function is of global smoothness level $1$ or $\infty$, so the LSE can adapt, if at all possible, to limited global structures. In fact, piecewise constancy is the only known global structure to which the LSE is confirmed to adapt, cf. [CGS18, HWCS18]. 2. (D2)

The LSE does not adapt at the optimal rate to constant signals when $d\geq 3$: it is shown in [HWCS18] that the LSE adapts to the global constant structures at a strictly sub-minimax rate $n^{-1/d}$ (up to logarithmic factors) in $L_{2}$-type losses.

The reasons for these limitations, however, lie in very different places: the drawback in (D1) is due to the perspective of considering global adaptation, while the drawback in (D2) is due to the use of the LSE.

In view of these limitations, in this paper we consider the local behavior of the following alternative max-min block estimator originally proposed by [FLN17]: for any $x_{0}\in[0,1]^{d}$,

[TABLE]

Here $[x_{u},x_{v}]=\{x\in\mathbb{R}^{d}:x_{u}\leq x\leq x_{v}\}$, $\bar{Y}|_{A}$ is the average of $\{Y_{i}:X_{i}\in A\}$ as in (1.6), and for any $x,y\in\mathbb{R}^{d}$, $x\leq y$ if and only if $x_{j}\leq y_{j}$ for all $1\leq j\leq d$, and the similar definition applies to $\geq$. It is easy to see that $\hat{f}_{n}\in\mathcal{F}_{d}$ and is tuning-free. The computation for (1.4) is exact and requires at most $\mathcal{O}(n^{2})$ for each design point, so the total computational complexity is at most $\mathcal{O}(n^{3})$, independent of the dimension $d$.

The max-min block estimator (1.4) above is closely related to the LSE studied in [HWCS18], in the sense that the LSE also admits a max-min representation [RWD88], but with the rectangles $[x_{u},1],[0,x_{v}]$ replaced by all upper sets and lower sets containing $x_{0}$. Since the upper and lower sets reduce to intervals in dimension one, (1.4) coincides with the standard univariate isotonic LSE in $d=1$. The representation through upper and lower sets is also observed in a related monotone density estimation problem in $d=2$, cf. [Pol98].

The max-min representation gives one heuristic explanation for the difficulty of the LSE in the sense of (D2): the class of upper and lower sets is too large for the partial sum process to remain tight in the large sample limit as soon as $d\geq 3$ (cf, [Dud14]). On the other hand, using the smaller class of rectangles as in (1.4), it is shown in [DZ18] that (1.4) does adapt to constant signals at a nearly optimal parametric rate in all dimensions, as opposed to the slower rate $n^{-1/d}$ for the LSE, cf. (D2). For the same reason, it is hard to expect a limiting distribution theory for the LSE.

The main contribution of this paper is to develop a limit distribution theory for the max-min block estimator (1.4). We show that, the limiting distribution of $\hat{f}_{n}$, depending on the local structure of $f_{0}$ at $x_{0}$, takes the following general form: Suppose $f_{0}$ admits vanishing derivatives up to order $\alpha_{k}$ along the $k$-th dimension ($k=1,\ldots,d$) at a fixed point $x_{0}\in(0,1)^{d}$, and the errors $\{\xi_{i}\}$ have variance $\sigma^{2}$. Then

[TABLE]

Here $\mathcal{D}_{\ast}$ and $n_{\ast}$, determined by the value of $(\alpha_{1},\ldots,\alpha_{d})$ and the design of $\{X_{i}\}$, are the set of all ‘effective dimensions’ and the size of ‘effective samples’ that drive the asymptotic limiting distribution, the exact meaning of which will be clarified in Section 2. The dependence of the limiting distribution $\mathbb{C}(f_{0},x_{0})$ on the local properties of $f_{0}$ at $x_{0}$ cannot be in general expressed by a simple factor, due to possible existence of non-zero mixed derivatives of critical order $\bm{j}=(j_{1},\ldots,j_{d})$ satisfying $\sum_{k=1}^{d}j_{k}/\alpha_{k}=1$ and $\lVert\bm{j}\rVert_{0}>1$. However, in situations where $\{\alpha_{k}\}$ are relative primes to each other (so that any such index vector $\bm{j}$ must have $\lVert\bm{j}\rVert_{0}=1$), or all mixed derivatives of $f_{0}$ of the critical order vanish at $x_{0}$, the limiting distribution $\mathbb{C}(f_{0},x_{0})$ can be represented in a similar form as in (1.1)-(1.2), namely

[TABLE]

Here $K(f_{0},x_{0})$ is a constant depending on the local structure of the regression function $f_{0}$ at $x_{0}$ to be specified in Section 2, and $\mathbb{D}_{\bm{\alpha}}$ is the non-standard limiting distribution playing the similar role as the Chernoff distribution $\mathbb{Z}$ in univariate problems.

One important and canonical setting for (1.5) is the following: Suppose (i) $f_{0}$ depends only through its first $s$ coordinates ($0\leq s\leq d$), and all non-trivial first-order partial derivatives of $f_{0}$ are non-vanishing at $x_{0}$: $\partial_{k}f_{0}(x_{0})>0,1\leq k\leq s$, and (ii) the design points $\{X_{i}\}$ are either of a balanced fixed lattice design (see Section 2 for a precise definition) or a random design with uniform distribution on $[0,1]^{d}$. In this setting, (1.5) reduces to:

[TABLE]

When $s=d=1$, we recover the familiar limit distribution theory for univariate isotonic least squares estimator.

The limit theory in (1.5), as we will see in Section 2, implies that the max-min block estimator (1.4) automatically adapts to the local smoothness structures and the intrinsic dimension of $f_{0}$. The local adaptation is in similar spirit to [BRW09, CW16, Wri81], who showed that univariate shape-restricted MLEs/LSEs adapt to local smoothness of the truth. It should be emphasized here that local smoothness to which adaptation occurs specifically refers to the number of vanishing (partial) derivatives. A distinct feature for the max-min block estimator (1.4) here is that both (i) the local rate of convergence, i.e. $n_{\ast}^{\frac{1}{2+\sum_{k\in\mathcal{D}_{\ast}}\alpha_{k}^{-1}}}$, and (ii) the dependence on $\{f_{0},x_{0}\}$ whenever explicit, i.e. the constant $K(f_{0},x_{0})$ in the limit distribution, are optimal in a local asymptotic minimax sense for all possible local smoothness levels. So in this sense the limit distribution theory for the max-min block estimator (1.4) in the form of (1.5) is the best one can hope for in the problem of multiple isotonic regression.

Another interesting consequence of (1.5) and its local asymptotic minimaxity is that the optimal local rates of convergence are in general not the same in fixed lattice and random designs. In fact, the local rate in the fixed lattice design case is no slower than that in the random design case, and can be much faster when (a) the local smoothness levels of the isotonic regression function, or (b) the sizes of the lattice, differ substantially along different dimensions. The reason for the discrepancy in the local rates can be attributed to the fact that significant imbalance in (a) or (b) screens out dimensions with ‘low regularity’ that do not contribute to the asymptotics in the fixed lattice design case. Here dimensions with ‘low regularity’, loosely speaking, refer to those with low smoothness levels in (a), and to those with sparsely spaced design points in (b).

The proof of the limit theory (1.5) in general dimensions is significantly more challenging than its univariate counterpart. Indeed, thanks to the ‘switching relation’ put forward by [Gro85], it is now well understood that limiting distributions for various univariate monotonicity shape constrained estimators can be obtained via the argmax continuous mapping theory, upon a proper one-sided localization (typically) on the order of a cubic-root rate (cf. [vdVW96]). In contrast, in the multiple isotonic regression problem we consider here, the key step in the proof is a two-sided localization technique, where the stochastic orders of the length of all sides of the rectangle over which the max-min block estimator (1.4) takes average, need to be estimated sharply both from above and below. These estimates bring about substantial technical challenges as opposed to univariate problems.

Finally we mention the work of [DZ18], in which global risk bounds in $L_{q}$ norms for the max-min block estimator (1.4) are thoroughly studied. Risk bounds in global metrics, as already mentioned in (D1), have a limited scope of structures for adaptation due to the strict global smoothness of the isotonic functions. Our local limit distribution theory (1.5) can therefore also be viewed as a further step in understanding the adaptive behavior of the max-min block estimator (1.4) to a rich class of structures that are exhibited only through local properties of the isotonic regression function.

The rest of the paper is organized as follows. In Section 2, we present the limit distribution theory (1.5) for the max-min block estimator (1.4), and discuss its many implications. In Section 3, we establish a local asymptotic minimax lower bound, showing the information-theoretic optimality of the limit theorem (1.5). Due to the highly technical nature of the proofs, Section 4 is devoted to an outline of the main ideas in the proofs. Section 5 concludes the paper with a brief discussion. All the proof details are presented in Sections 6-9.

1.2. Notation

For a real-valued measurable function $f$ defined on $(\mathcal{X},\mathcal{A},P)$, \lVert f\rVert_{L_{p}(P)}\equiv\lVert f\rVert_{P,p}\equiv\big{(}P\lvert f\rvert^{p})^{1/p} denotes the usual $L_{p}$-norm under $P$, and $\lVert f\rVert_{\infty}\equiv\sup_{x\in\mathcal{X}}\lvert f(x)\rvert$. Let $(\mathcal{F},\lVert\cdot\rVert)$ be a subset of the normed space of real functions $f:\mathcal{X}\to\mathbb{R}$. For $\varepsilon>0$ let $\mathcal{N}(\varepsilon,\mathcal{F},\lVert\cdot\rVert)$ be the $\varepsilon$-covering number of $\mathcal{F}$; see page 83 of [vdVW96] for more details.

For the regression model (1.3), for any $A\subset[0,1]^{d}$, define

[TABLE]

where $n_{A}\equiv\lvert\{i:X_{i}\in A\}\rvert$.

For two real numbers $a,b$, $a\vee b\equiv\max\{a,b\}$ and $a\wedge b\equiv\min\{a,b\}$. For $x\in\mathbb{R}^{d}$, let $\lVert x\rVert_{p}$ denote its $p$-norm $(0\leq p\leq\infty)$. For any $x,y\in\mathbb{R}^{d}$, let $[x,y]\equiv\prod_{k=1}^{d}[x_{k}\wedge y_{k},x_{k}\vee y_{k}]$, $xy\equiv(x_{k}y_{k})_{k=1}^{d}$, and $x\wedge(\vee)y\equiv(x_{k}\wedge(\vee)y_{k})_{k=1}^{d}$. For $\ell_{1},\ell_{2}\in\{1,\ldots,d\}$, we let $\bm{1}_{[\ell_{1}:\ell_{2}]}\in\mathbb{R}^{d}$ be such that $(\bm{1}_{[\ell_{1}:\ell_{2}]})_{k}=\bm{1}_{\ell_{1}\leq k\leq\ell_{2}}$, and $\bm{1}\equiv\bm{1}_{[1:d]}$ for simplicity. $C_{x}$ will denote a generic finite constant that depends only on a generic quantity $x$, whose numeric value may change from line to line unless otherwise specified. $a\lesssim_{x}b$ and $a\gtrsim_{x}b$ mean $a\leq C_{x}b$ and $a\geq C_{x}b$ respectively, and $a\asymp_{x}b$ means $a\lesssim_{x}b$ and $a\gtrsim_{x}b$ [$a\lesssim b$ means $a\leq Cb$ for some absolute constant $C$]. $\mathcal{O}_{\mathbf{P}}$ and $\mathfrak{o}_{\mathbf{P}}$ denote the usual big and small O notation in probability. $\rightsquigarrow$ is reserved for weak convergence. For two integers $k_{1}>k_{2}$, we interpret $\sum_{k=k_{1}}^{k_{2}}\equiv 0,\prod_{k=k_{1}}^{k_{2}}\equiv 1$. We also interpret $(\infty)^{-1}\equiv 0,0/0\equiv 0$.

For $f:\mathbb{R}^{d}\to\mathbb{R}$, and $k\in\{1,\ldots,d\}$, $\alpha_{k}\in\mathbb{Z}_{\geq 1}$, let $\partial_{k}^{\alpha_{k}}f(x)\equiv\frac{d^{\alpha_{k}}}{dx_{k}^{\alpha_{k}}}f(x)$. For a multi-index $\bm{j}=(j_{1},\ldots,j_{d})\in\mathbb{Z}_{\geq 0}^{d}$, let $\partial^{\bm{j}}\equiv\partial_{1}^{j_{1}}\cdots\partial_{d}^{j_{d}}$, and $\bm{j}!\equiv j_{1}!\cdots j_{d}!$ and $x^{\bm{j}}\equiv x_{1}^{j_{1}}\ldots x_{d}^{j_{d}}$ for $x\in\mathbb{R}^{d}$. For $\bm{\alpha}=(\alpha_{1},\ldots,\alpha_{d})\in\mathbb{Z}_{\geq 1}^{d}$ in Assumption A below, i.e. for some $0\leq s\leq d$, $1\leq\alpha_{1},\ldots,\alpha_{s}<\infty=\alpha_{s+1}=\ldots=\alpha_{d}$, let $J(\bm{\alpha})$ (resp. $J_{\ast}(\bm{\alpha})$) be the set of all $\bm{j}=(j_{1},\ldots,j_{d})\in\mathbb{Z}_{\geq 0}^{d}$ satisfying $0<\sum_{k=1}^{s}j_{k}/\alpha_{k}\leq 1$ (resp. $\sum_{k=1}^{s}j_{k}/\alpha_{k}=1$) and $j_{k}=0$ for $s+1\leq k\leq d$, and let $J_{0}(\bm{\alpha})\equiv J(\bm{\alpha})\cup\{\bm{0}\}$. We often write $J=J(\bm{\alpha})$, $J_{\ast}=J_{\ast}(\bm{\alpha})$ and $J_{0}=J_{0}(\bm{\alpha})$ if no confusion arises. The set $J,J_{\ast}$ will play a crucial role below in determining $\bm{j}$’s for which $\partial^{\bm{j}}f_{0}(x_{0})$ can be non-zero under Assumption A; cf. Lemma 1.

2. Limit distribution theory

2.1. Assumptions

We first state the assumptions on the local smoothness of $f_{0}$ at the point of interest $x_{0}\in(0,1)^{d}$ and the intrinsic dimension of $f_{0}$.

Assumption A*.*

$f_{0}$ is coordinate-wise nondecreasing (i.e. $f_{0}\in\mathcal{F}_{d}$), and is $\bm{\alpha}$-smooth at $x_{0}$ with intrinsic dimension $s$, $\bm{\alpha}=(\alpha_{1},\ldots,\alpha_{d})$ with integers $1\leq\alpha_{1},\ldots,\alpha_{s}<\infty=\alpha_{s+1}=\ldots=\alpha_{d}$, $0\leq s\leq d$, in the sense that $\partial_{k}^{j_{k}}f_{0}(x_{0})=0$ for $1\leq j_{k}\leq\alpha_{k}-1$ and $\partial_{k}^{\alpha_{k}}f_{0}(x_{0})\neq 0$, $1\leq k\leq s$, and in rectangles of the form $\cap_{k=1}^{d}\{\lvert(x-x_{0})_{k}\rvert\leq L_{0}\cdot(r_{n})_{k}\}$, $r_{n}=(\omega_{n}^{1/\alpha_{1}},\ldots,\omega_{n}^{1/\alpha_{d}})$ with $\omega_{n}>0$, the Taylor expansion of $f_{0}$ satisfies for all $L_{0}>0$,

[TABLE]

Assumption A concerns the local smoothness of $f_{0}$ at a fixed point $x_{0}$, allowing for potentially different local smoothness levels along different coordinates $\{1,\ldots,s\}$. The Taylor expansion, which includes all terms of order $\omega_{n}$ or larger in a small hyper-rectangle, interestingly features different rates $\omega_{n}^{1/\alpha_{k}}$ in different dimensions in the $x$-domain. This is quite different from the Taylor expansion in Euclidean balls which includes all terms with $\|\bm{j}\|_{1}\leq\alpha$ for a certain smoothness index $\alpha$. Our expansion has the prescribed convergence rate if $f_{0}$ is locally $C^{\max_{1\leq k\leq s}\alpha_{k}}$ at $x_{0}$ and depends only through its first $s$ coordinates. Note that Assumption A is interesting mostly from a local perspective. Indeed, if this condition holds for all $x_{0}\in(0,1)^{d}$ with some $1\leq\alpha_{1},\ldots,\alpha_{s}<\infty$, then we must have $\alpha_{1}=\ldots=\alpha_{s}=1$.

Now we consider a few examples that satisfy Assumption A with different values of $\bm{\alpha}$’s. In the following examples we consider $d=2$ and $x_{0}=(1/2,1/2)$ unless otherwise specified.

Example 1**.**

Let $f_{0}^{(1)}(x_{1},x_{2})=x_{1}+x_{2}$. Then $\alpha_{1}=\alpha_{2}=1$.

Example 2**.**

Let $f_{0}^{(2)}(x_{1},x_{2})=x_{1}$. Then $s=1$ with $\alpha_{1}=1,\alpha_{2}=\infty$.

Example 3**.**

Let $f_{0}^{(3)}(x_{1},x_{2})=(x_{1}+x_{2})\bm{1}_{0\leq x_{1}\leq 1/4}+8x_{1}\cdot\bm{1}_{1/4<x_{1}<3/4}+8(x_{1}+x_{2})\bm{1}_{3/4\leq x_{1}\leq 1}$. Then $s=1$ with $\alpha_{1}=1,\alpha_{2}=\infty$ for $x_{0}\in(1/4,3/4)\times(0,1)$, and $s=2$ with $\alpha_{1}=\alpha_{2}=1$ for $x_{0}\in(0,1/4)\times(0,1)\cup(3/4,1)\times(0,1)$.

Example 2 is a canonical example for which the regression function is globally of intrinsic dimension 1, while in Example 3 the function can be locally of intrinsic dimension 1 in the strip $(1/4,3/4)\times(0,1)$.

Example 4**.**

Let $f_{0}^{(4)}(x_{1},x_{2})=(x_{1}-1/2)^{3}+(x_{2}-1/2)^{3}$. Then $\alpha_{1}=3,\alpha_{2}=3$.

Example 5**.**

Let $f_{0}^{(5)}(x_{1},x_{2})=(x_{1}-1/2)^{3}+(x_{1}-1/2)^{2}(x_{2}-1/2)+(x_{1}-1/2)(x_{2}-1/2)^{2}+(x_{2}-1/2)^{3}$. Then $\alpha_{1}=3,\alpha_{2}=3$.

Example 4 and Example 5 both share the same local smoothness level $\bm{\alpha}=(3,3)$, but are quite different in that for $f_{0}^{(4)}$ all mixed derivatives vanish, while for $f_{0}^{(5)}$ certain mixed derivatives do no vanish: $\partial^{\bm{j}}f_{0}^{(5)}(x_{0})\neq 0$ for $\bm{j}\in\{(1,2),(2,1)\}$.

Lemma 1**.**

The following statements hold:

1. (1)

Suppose Assumption A holds. $\alpha_{k}$ must be odd and $\partial_{k}^{\alpha_{k}}f_{0}(x_{0})>0$ for $1\leq k\leq s$; 2. (2)

Suppose Assumption A holds. Any mixed derivative of the form $\partial^{\bm{j}}f_{0}(x_{0})$, $\bm{0}\neq\bm{j}\in J\setminus J_{\ast}$, vanishes at $x_{0}$, and thus for some $\varepsilon_{1}>0$, $L_{1}>0$ depending only on $f_{0},x_{0}$,

[TABLE] 3. (3)

Let $J_{1}\equiv\{\bm{j}\in J_{\ast}:\lVert\bm{j}\rVert_{0}>1\}$. Then $J_{1}=\emptyset$ if and only if $\lvert J_{\ast}\rvert=s$, if and only if $\{\alpha_{k}\}_{k=1}^{s}$ is a set of relative primes, i.e. the greatest common divisor of $\{\alpha_{k_{1}},\alpha_{k_{2}}\}$ is 1 for all $1\leq k_{1}<k_{2}\leq s$; 4. (4)

When $\bm{\alpha}$ is such that $J_{1}\neq\emptyset$, there exists some $f\in\mathcal{F}_{d}$ for which $f$ satisfies Assumption A with $\bm{\alpha}$, but $\partial^{\bm{j}}f(x_{0})\neq 0$ for some $\bm{j}\in J_{1}$.

Proof.

See Section 8. ∎

Lemma 1 reveals an important and unique feature of multiple isotonic functions compared with smooth functions: If $f_{0}$ satisfies the ‘marginal smoothness’ Assumption A with $\bm{\alpha}=(\alpha_{1},\ldots,\alpha_{d})$ at $x_{0}$, then the only possible non-zero mixed derivatives $\partial^{\bm{j}}f_{0}(x_{0})$ in the Taylor expansion must have critical order $\bm{j}\in J_{\ast}$ satisfying $\sum_{k=1}^{s}j_{k}/\alpha_{k}=1$. Such possible non-zero mixed derivatives cannot be ruled out under Assumption A as soon as certain pair of $\{\alpha_{k}\}$ has a non-trivial common divisor. The importance of such a feature lies in the fact that these mixed derivatives $\{\partial^{\bm{j}}f_{0}(x_{0}):\bm{j}\in J_{\ast}\}$ contribute to the convergence rate of the same order as the marginal derivatives $\{\partial_{k}^{\alpha_{k}}f_{0}(x_{0})\}$, in rectangles of the form $\cap_{k=1}^{d}\{\lvert(x-x_{0})_{k}\rvert\leq L_{0}\cdot(r_{n})_{k}\}$, $r_{n}=(\omega_{n}^{1/\alpha_{1}},\ldots,\omega_{n}^{1/\alpha_{d}})$ with $\omega_{n}\searrow 0$. Hence adaptation of the max-min estimator (1.4) to marginal smoothness levels—which only uses marginal information in rectangles—becomes possible.

Next we state the assumptions on the design of the covariates.

Assumption B*.*

The design points $\{X_{i}\}_{i=1}^{n}$ satisfy either of the following:

• •

(Fixed design) $\{X_{i}\}$’s follow a $\bm{\beta}$-fixed lattice design: there exist some $\{\beta_{1},\ldots,\beta_{d}\}\subset(0,1)$ with $\sum_{k=1}^{d}\beta_{k}=1$ such that $x_{0}\in\{X_{i}\}_{i=1}^{n}=\prod_{k=1}^{d}\{x_{1,k},\ldots,x_{n_{k},k}\}$, where $\{x_{1,k},\ldots,x_{n_{k},k}\}$ are equally spaced in $[0,1]$ (i.e. $\lvert x_{j,k}-x_{j+1,k}\rvert=1/n_{k}$ for all $j=1,\ldots,n_{k}-1$) and $n_{k}=\left\lfloor n^{\beta_{k}}\right\rfloor$.

• •

(Random design) $\{X_{i}\}$’s follow i.i.d. random design with law $P$ independent of $\{\xi_{i}\}$’s. The Lebesgue density $\pi$ of $P$ is bounded away from [math] and $\infty$ on $[0,1]^{d}$ and is continuous over an open set containing the region \big{\{}\big{(}(x_{0})_{1},\ldots,(x_{0})_{s},x_{s+1},\ldots,x_{d}\big{)}:0\leq x_{k}\leq 1,s+1\leq k\leq d\big{\}}.

In the fixed lattice design case, we use $\beta_{k}$ to control the size of the lattice in dimension $k$. A balanced fixed lattice design refers to the special case with $\beta_{k}=1/d$ for all $k=1,\ldots,d$. In the random design case, the continuity of the density $\pi$ is imposed over the region where asymptotics take place.

We choose without loss of generality the index $\{1,\ldots,d\}$ such that

[TABLE]

This requirement facilitates the statement of our main Theorem 1 below. Otherwise we may find some permutation $\tau$ of $\{1,\ldots,d\}$ for which $\alpha_{\tau(1)}\beta_{\tau(1)}\leq\ldots\leq\alpha_{\tau(d)}\beta_{\tau(d)}$, and consider the coordinate-wise nondecreasing function $\tilde{f}_{0}(x_{1},\ldots,\ldots,x_{d})\equiv f_{0}(x_{\tau(1)},\ldots,x_{\tau(d)})$. Such a reparametrization is compatible with Assumption A since $\alpha_{k}\beta_{k}=\infty$ if and only if $\alpha_{k}=\infty$.

2.2. Limit distribution theory

Let $x_{0}\in(0,1)^{d}$. Let $\pi(x)\equiv 1$ in the $\bm{\beta}$-fixed lattice design case and $\pi(x)\equiv\mathrm{d}P/(\mathrm{d}x_{1}\cdots\mathrm{d}x_{d})$ be the Lebesgue density of $P$ in the random design case. For any $0\leq s\leq d$, and $h_{1},h_{2}\in\mathbb{R}_{\geq 0}^{d}$ such that $(h_{1})_{k}\leq(x_{0})_{k},(h_{2})_{k}\leq(1-x_{0})_{k}$ for all $s+1\leq k\leq d$, let

[TABLE]

and $\mathcal{I}^{[d+1:d]}_{\pi}\equiv\pi(x_{0})$. The integration above is carried out over the region $\{(x_{0}-h_{1})_{k}\leq x_{k}\leq(x_{0}+h_{2})_{k}:s+1\leq k\leq d\}\subset[0,1]^{d-s}$ with the integrand given by the Lebesgue density $\pi$ of the design distribution $P$. In the $\bm{\beta}$-fixed lattice design case, $\mathcal{I}^{[s+1:d]}_{\pi}(h_{1},h_{2})=\prod_{k=s+1}^{d}(h_{1}+h_{2})_{k}$.

Let $\kappa_{\ast},n_{\ast}$ be defined by

[TABLE]

Let the limit process $\mathbb{C}(f_{0},x_{0})$ be defined by

[TABLE]

where $\mathbb{G}$ is a centered Gaussian process defined on $\mathbb{R}^{d}_{\geq 0}\times\mathbb{R}^{d}_{\geq 0}$ with the following covariance structure: for any $(h_{1},h_{2}),(h_{1}^{\prime},h_{2}^{\prime})$,

[TABLE]

and

[TABLE]

Furthermore, let $\mathbb{D}_{\bm{\alpha}}$ be defined by

[TABLE]

With these definitions, we are now in position to state the main result of this paper.

Theorem 1**.**

Suppose Assumptions A-B hold, and the errors $\{\xi_{i}\}$ are i.i.d. mean-zero with finite variance $\mathbb{E}\xi_{1}^{2}=\sigma^{2}<\infty$ (and are independent of $\{X_{i}\}$ in the random design case). With $\kappa_{\ast},n_{\ast}$ defined in Table 1, we have the following local rate of convergence:

[TABLE]

If $\kappa_{\ast}$ is uniquely defined, with $\mathbb{C}(f_{0},x_{0})$ defined in (2.2), the following limit theory holds:

[TABLE]

Furthermore, if either $\{\alpha_{k}\}$ is a set of relative primes or all mixed derivatives of $f_{0}$ vanish at $x_{0}$ in $J_{\ast}$, then

[TABLE]

where K(f_{0},x_{0})=\big{\{}\prod_{k=\kappa_{\ast}}^{s}\big{(}{\partial_{k}^{\alpha_{k}}f_{0}(x_{0})}/{(\alpha_{k}+1)!}\big{)}^{1/\alpha_{k}}\big{\}}^{\frac{1}{2+\sum_{k=\kappa_{\ast}}^{s}\alpha_{k}^{-1}}} and $\mathbb{D}_{\bm{\alpha}}$ is defined in (2.3).

Proof.

See Section 6. ∎

Remark 1*.*

A few technical remarks:

1. (1)

In the $\bm{\beta}$-fixed lattice design case where $\pi(x)=1$ is used to calculate $\mathcal{I}^{[s+1:d]}_{\pi}(h_{1},h_{2})$, the covariance structure of $\mathbb{G}$ is simpler:

[TABLE]

and can be represented as follows: let $d_{\ast}\equiv d-\kappa_{\ast}+1$, and let $\{\mathbb{B}_{i}:i\in\{1,2\}^{d_{\ast}}\}$ be independent Brownian sheets on $\mathbb{R}_{\geq 0}^{d_{\ast}}$. For any $h_{1},h_{2}\in\mathbb{R}_{\geq 0}^{d}$, let \bar{\mathbb{G}}(h_{1},h_{2})\equiv\sum_{i\in\{1,2\}^{d_{\ast}}}\mathbb{B}_{i}\big{(}(h_{i_{1}})_{\kappa_{\ast}},\ldots,(h_{i_{d_{\ast}}})_{d}\big{)}. Then $\mathbb{G}(\cdot,\cdot)=_{d}\bar{\mathbb{G}}(\cdot,\cdot)$. A similar representation holds in the random design case for $s=d$. 2. (2)

$\mathbb{C}(f_{0},x_{0})$ has at least a sub-gaussian tail and hence admits moments of all orders (cf. Lemma 4). 3. (3)

When either $\{\alpha_{k}\}$ is a set of relative primes or all mixed derivatives of $f_{0}$ vanish at $x_{0}$ in $J_{\ast}$, the following self-similarity property of the process $\mathbb{G}(\cdot,\cdot)$ is essential for the representation $\mathbb{C}(f_{0},x_{0})=_{d}K(f_{0},x_{0})\cdot\mathbb{D}_{\bm{\alpha}}$: for $\gamma\in\mathbb{R}_{\geq 0}^{d}$ with $\gamma_{1}=\ldots=\gamma_{\kappa_{\ast}-1}=0,\gamma_{\kappa_{\ast}}\ldots,\gamma_{s},\gamma_{s+1}=\ldots=\gamma_{d}=1$, \mathbb{G}(\gamma\cdot,\gamma\cdot)=_{d}\big{(}\prod_{k=\kappa_{\ast}}^{s}\gamma_{k}\big{)}^{1/2}\cdot\mathbb{G}(\cdot,\cdot). 4. (4)

$\mathbb{D}_{\bm{\alpha}}$ (and $\mathbb{C}(f_{0},x_{0})$) can be represented by sup-inf over the summation of a stochastic term plus a non-random drift term, similar to that of the Chernoff distribution; see (2.6) below for an explicit derivation of $\mathbb{D}_{1}$ being the Chernoff distribution. 5. (5)

Although implicit in notation, $\mathbb{D}_{\bm{\alpha}}$ can depend on $x_{0}$ through $\mathcal{I}_{\pi}^{[s+1:d]}$. However, such dependence disappears under (i) the fixed lattice design and (ii) the random design with uniform distribution. For general distributions $P$ in the random design case, if $s=d$ and $\mathbb{C}(f_{0},x_{0})=_{d}K(f_{0},x_{0})\cdot\mathbb{D}_{\bm{\alpha}}$, the dependence of $x_{0}$ within $\mathbb{D}_{\bm{\alpha}}$ can be assimilated into the constant. In fact, by taking $\sigma$ to be $\sigma/\sqrt{\pi(x_{0})}$ in (6.17), we have:

[TABLE]

where $\mathbb{G}$ is the Gaussian process with the covariance structure (2.4).

Theorem 1 shows that the max-min block estimator (1.4) adapts to the local smoothness levels $\{\alpha_{k}\}$ and the intrinsic dimension $s$ of the isotonic regression function $f_{0}$, in both the fixed lattice and random design settings. One particularly interesting consequence of the above theorem is that the adaptive local rates for the fixed lattice and random design cases are in general not the same. Indeed,

[TABLE]

so that $\omega_{n}^{\mathrm{fixed}}\leq\omega_{n}^{\mathrm{random}}$, i.e., the local rate in the fixed lattice design case is no slower than that in the random design case.

The following proposition gives an equivalent definition of $\kappa_{\ast}$ in the fixed lattice design case in Theorem 1.

Proposition 1**.**

The following are equivalent under (2.1):

1. (1)

The maximizer of $\ell\mapsto\frac{\sum_{k=\ell}^{d}\beta_{k}}{2+\sum_{k=\ell}^{s}\alpha_{k}^{-1}}$ is unique and $\kappa_{\ast}=\arg\max\limits_{1\leq\ell\leq d}\frac{\sum_{k=\ell}^{d}\beta_{k}}{2+\sum_{k=\ell}^{s}\alpha_{k}^{-1}}$. 2. (2)

For any $1\leq\ell\leq d$, $\frac{\alpha_{\ell}^{-1}}{2+\sum_{k=\ell}^{s}\alpha_{k}^{-1}}\neq\frac{\beta_{\ell}}{\sum_{k=\ell}^{d}\beta_{k}}$, and \kappa_{\ast}=\min\big{\{}1\leq\ell\leq d:\frac{\alpha_{\ell}^{-1}}{2+\sum_{k=\ell}^{s}\alpha_{k}^{-1}}<\frac{\beta_{\ell}}{\sum_{k=\ell}^{d}\beta_{k}}\big{\}}=\min\{1\leq\ell\leq d:(\omega_{n}^{(\ell)})^{1/\alpha_{\ell}}n^{\beta_{\ell}}>1\}. Here $\omega_{n}^{(\ell)}\equiv n^{-\frac{\sum_{k=\ell}^{d}\beta_{k}}{2+\sum_{k=\ell}^{s}\alpha_{k}^{-1}}}$ is the unique solution of the fixed-point equation

[TABLE]

Proof.

By algebra, for any relationship $\sim$ in the set $\{<,\leq,>,\geq\}$, we have (i) $\frac{\sum_{k=\ell}^{d}\beta_{k}}{2+\sum_{k=\ell}^{s}\alpha_{k}^{-1}}\sim\frac{\sum_{k=\ell+1}^{d}\beta_{k}}{2+\sum_{k=\ell+1}^{s}\alpha_{k}^{-1}}$ if and only if (ii) $\frac{\beta_{\ell}}{\sum_{k=\ell}^{d}\beta_{k}}\sim\frac{\alpha_{\ell}^{-1}}{2+\sum_{k=\ell}^{s}\alpha_{k}^{-1}}$ if and only if (iii) 2\sim\sum_{k=\ell}^{d}\beta_{k}\big{(}\frac{1}{\alpha_{\ell}\beta_{\ell}}-\frac{1}{\alpha_{k}\beta_{k}}\big{)}\equiv\psi(\ell). Under the ordering (2.1), $\ell\mapsto\psi(\ell)$ is non-increasing, so the statement (1) holds if and only if $\sim$ is taken as $<$ for all $1\leq\ell\leq\kappa_{\ast}-1$ and as $>$ for $\kappa_{\ast}\leq\ell\leq d$ in (i), if and only if the same $\sim$ are taken in (ii), if and only if the statement (2) holds. ∎

Proposition 1 (2) shows that $\kappa_{\ast}$ can be determined by a sequence of bias-variance equations in (2.5). This gives an interesting interpretation of the quantities $\kappa_{\ast},n_{\ast}$ in the $\bm{\beta}$-fixed lattice design case:

• •

$\kappa_{\ast}(\leq(s+1)\wedge d)$ can be viewed as a ‘critical dimension’ in the sense that samples in dimensions $\{1,\ldots\kappa_{\ast}-1\}$ do not contribute in the asymptotics of $\hat{f}_{n}$. In other words, the limit distribution of $\hat{f}_{n}$ is fully driven by samples in dimensions $\{\kappa_{\ast},\ldots,d\}$. The uniqueness of the maximizer of $\ell\mapsto\frac{\sum_{k=\ell}^{d}\beta_{k}}{2+\sum_{k=\ell}^{s}\alpha_{k}^{-1}}$ gives a well-defined $\kappa_{\ast}$, and therefore the limiting distribution $\mathbb{C}(f_{0},x_{0})$.

• •

$n_{\ast}=n^{\sum_{k=\kappa_{\ast}}^{d}\beta_{k}}$ can be viewed as the ‘effective sample size’ over the effective dimensions $\{\kappa_{\ast},\ldots,d\}$ for the asymptotics of $\hat{f}_{n}$.

In contrast, in the random design case the ‘critical dimension’ $\kappa_{\ast}$ is always $\kappa_{\ast}=1$, as long as the Lebesgue density of $P$ is suitably regular at $x_{0}$. In this setting all dimensions $\{1,\ldots,d\}$ are effective, and the ‘effective sample size’ is simply $n_{\ast}=n$.

The local rate of convergence in Theorem 1 can now be interpreted very naturally: the exponent for the ‘effective sample size’ $n_{\ast}$, namely, $\frac{1}{2+\sum_{k=\kappa_{\ast}}^{s}\alpha_{k}^{-1}}$ becomes the local smoothness along ‘effective dimensions’ $\{\kappa_{\ast},\ldots,d\}$ (note that $\alpha_{k}^{-1}=0$ for $s+1\leq k\leq d$).

Remark 2*.*

In the special case that $f_{0}$ is globally flat, i.e. $f_{0}\equiv c$ for some $c\in\mathbb{R}$, we have $\alpha_{1}=\ldots=\alpha_{d}=\infty$, and therefore the local rate of convergence for the max-min block estimator (1.4) is parametric $\mathcal{O}_{\mathbf{P}}(n^{-1/2})$. When $f_{0}$ is locally flat, it is shown by [CD99] (see also [Gro83]) that in the closely related univariate monotone density estimation problem, the Grenander estimator converges at a parametric rate, with a limiting distribution involving the maximal interval contained in the flat region. In the multivariate case, the shape for locally flat regions can be quite complicated. For example, for $d=2$ and any upper set $U\subset[1/2,1]^{2}$, consider $f_{0}\equiv\bm{1}_{U}$. Then the local rate of convergence for the max-min block estimator (1.4) is still parametric at, say, $(1/4,1/4)$, but the limiting distribution would depend crucially on the exact shape of $U$. It is an interesting open problem to characterize all possible locally flat regions and derive the corresponding limiting distributions for the max-min block estimator (1.4).

2.3. Comparison of local rates

In this section, we make comparisons of the local rates in different fixed lattice and random designs. As will be seen below, the discrepancy of the local rates appears when either the local smoothness levels of the isotonic regression function, or the sizes of the lattice differ substantially along different dimensions.

2.3.1. Difference in local rates due to imbalanced local smoothness levels

Consider the case where the local smoothness levels are imbalanced with $\alpha_{1}=\ldots=\alpha_{s}=\alpha,\alpha_{s+1}=\ldots=\alpha_{d}=\infty$ for some $\alpha\geq 1,1\leq s<d$, while the sizes of the lattice are balanced with $\beta_{1}=\ldots=\beta_{d}=1/d$. By Theorem 1, we have the following corollary:

Corollary 1**.**

Suppose that the assumptions in Theorem 1 hold with $\alpha_{1}=\ldots=\alpha_{s}=\alpha\geq 1,\alpha_{s+1}=\ldots=\alpha_{d}=\infty$ for some $1\leq s<d$, and $\beta_{1}=\ldots=\beta_{d}=1/d$. Then in the fixed lattice design case,

[TABLE]

In the random design case,

[TABLE]

The local rates can be written more compactly:

• •

(Fixed design): \hat{f}_{n}(x_{0})-f_{0}(x_{0})=\mathcal{O}_{\mathbf{P}}\big{(}n^{-\max\{\frac{1}{2+s/\alpha},\frac{1}{2}\cdot\frac{d-s}{d}\}}\big{)}.

• •

(Random design): $\hat{f}_{n}(x_{0})-f_{0}(x_{0})=\mathcal{O}_{\mathbf{P}}(n^{-\frac{1}{2+s/\alpha}})$.

Below we consider two scenarios according to the phase transition boundary $\alpha=(d-s)/2$ given above in the balanced fixed lattice design case.

(Scenario 1: $\alpha>(d-s)/2$). In this case, $\kappa_{\ast}=1$ in the fixed lattice design case, so $\omega_{n}^{\mathrm{fixed}}=\omega_{n}^{\mathrm{random}}=n^{-\frac{1}{2+s/\alpha}}$. This includes the important case of $s=d$. In the special case where $d=1$ and $\alpha_{1}=1$, Corollary 1 reduces to the limit distribution theory for univariate isotonic regression: Suppose for simplicity we consider the fixed balanced lattice design, or the uniform random design. Then $\mathbb{C}(f_{0},x_{0})=_{d}K(f_{0},x_{0})\cdot\mathbb{D}_{1}=(f_{0}^{\prime}(x_{0})/2)^{1/3}\cdot\mathbb{D}_{1}$, where $\mathbb{D}_{1}$ is the well-known (rescaled) Chernoff distribution. To see this, with $\mathbb{B}$ denoting the standard two-sided Brownian motion starting at [math], we have (cf. Section 3.3 of [GJ14])

[TABLE]

It is also interesting to observe that for the most natural case $\alpha_{1}=\ldots=\alpha_{d}=1$, the local rate is $\mathcal{O}_{\mathbf{P}}(n^{-1/(2+d)})$. This local rate is, somewhat surprisingly, faster than the global minimax rate $\mathcal{O}(n^{-1/2d})$ in $L_{2}$ metric for $d\geq 3$, cf. [HWCS18]. The reason for this is that the global minimax rate in $L_{2}$ metric is dominated by the anti-chain structure of the multiple isotonic regression functions (cf. [HWCS18]), while the smoothness constraint rules out such a structure locally at a fixed point. To put the problem in other words, the global minimax rate in $L_{2}$ metric is too conservative in capturing the smoothness structure of the isotonic functions as soon as $d\geq 3$.

(Scenario 2: $\alpha<(d-s)/2$). In this case $\kappa_{\ast}=s+1>1$, so the local rate of convergence in the fixed lattice design case is much faster than that in the random design case: $\omega_{n}^{\mathrm{fixed}}\ll\omega_{n}^{\mathrm{random}}$.

Let us consider one concrete situation to better understand this phenomenon: $\alpha=1,s=1$ and $d>3$. The local rate is then $\mathcal{O}_{\mathbf{P}}(n^{-\frac{1}{2}\cdot\frac{d-1}{d}})$ in the fixed lattice design case, and is $\mathcal{O}_{\mathbf{P}}(n^{-1/3})$ in the random design case. Suppose for simplicity the regression function $f_{0}(x_{0})=f_{0}((x_{0})_{1},\ldots,(x_{0})_{d})=g_{0}((x_{0})_{1})$ for some one-dimensional nondecreasing function $g_{0}$. Consider fixed lattice and random design cases separately:

• •

In the fixed lattice design case, the oracle estimator first takes sample mean in dimensions $2$ to $d$, and then performs isotonic regression in dimension $1$ with reduced variance $\sigma_{1}^{2}\equiv\sigma^{2}/n^{(d-1)/d}$ and sample size $n_{1}=n^{1/d}$. However, as long as $d>3$, there are no longer large samples within the oracle bandwidth $(\sigma_{1}^{2}/n_{1})^{1/3}=(\sigma^{2}/n)^{1/3}\ll n^{-1/d}$ in dimension $1$ due to the smoothness. This means that the oracle estimator is simply the sample mean over dimensions $2$ to $d$ with a convergence rate $n^{-\frac{1}{2}\cdot\frac{d-1}{d}}$ when $d>3$. See the left panel of Figure 1.

• •

In the random design case, since the first coordinates of the design points are distinct with probability one, the oracle estimator is the one dimensional estimator with a bandwidth on the order of $n^{-1/3}$. This gives the usual convergence rate $n^{-1/3}$. See the right panel of Figure 1.

Corollary 1 with $\alpha=1,s=1,d>3$ can then be understood as saying that the max-min block estimator (1.4) mimics this oracle behavior in terms of the local rate of convergence, in both the fixed lattice and random design cases. In more general settings of Corollary 1, as soon as the local smoothness levels $\alpha<(d-s)/2$, the first $s$ dimensions are screened out in the fixed lattice design case, so the asymptotics only take place over pure noises in dimensions $\{s+1,\ldots,d\}$.

2.3.2. Difference in local rates due to imbalanced lattice sizes

Consider the case where the local smoothness levels are balanced with $\alpha_{1}=\ldots=\alpha_{d}=\alpha\geq 1$, while the sizes of the lattice are imbalanced with $\beta_{1}\leq\ldots\leq\beta_{d}$. Using $\sum_{k=1}^{d}\beta_{k}=1$, Theorem 1 and the equivalent definition of $\kappa_{\ast}$ in Proposition 1, we have the following.

Corollary 2**.**

Suppose that the assumptions in Theorem 1 hold with $\alpha_{1}=\ldots=\alpha_{d}=\alpha\geq 1$, and $\beta_{1}\leq\ldots\leq\beta_{d}$.

In the fixed lattice design case, suppose $\beta_{\ell}\neq\frac{1-\sum_{k=1}^{\ell-1}\beta_{k}}{2\alpha+d-\ell+1}$ for all $1\leq\ell\leq d$. Let \kappa_{\ast}\equiv\min\big{\{}1\leq\ell\leq d:\beta_{\ell}>\frac{1-\sum_{k=1}^{\ell-1}\beta_{k}}{2\alpha+d-\ell+1}\big{\}} and $d_{\ast}\equiv d-\kappa_{\ast}+1$. Then

[TABLE]

In the random design case,

[TABLE]

Alternatively, we may write the local rates more compactly:

• •

(Fixed design): \hat{f}_{n}(x_{0})-f_{0}(x_{0})=\mathcal{O}_{\mathbf{P}}\big{(}n^{-\max_{1\leq\ell\leq d}\frac{\sum_{k=\ell}^{d}\beta_{k}}{2+(d-\ell+1)/\alpha}}\big{)}.

• •

(Random design): $\hat{f}_{n}(x_{0})-f_{0}(x_{0})=\mathcal{O}_{\mathbf{P}}(n^{-\frac{1}{2+d/\alpha}})$.

The basic pattern for the phase transition phenomenon here is similar to the discussion in the previous section. The difference is that now it is the imbalance of the sizes of the lattice in different dimensions, rather than the local smoothness levels, that screens out dimensions with too sparsely spaced design points. See Figure 2 for the concrete phase transition in the line segment $\{\beta_{1}+\beta_{2}=1,\beta_{1}\leq\beta_{2}\}$ for $d=2$ and in the triangle $\{\beta_{1}+\beta_{2}+\beta_{3}=1,\beta_{1}\leq\beta_{2}\leq\beta_{3}\}$ for $d=3$.

Remark 3*.*

The improvement of the local rate in the fixed lattice design case over the random design case is strongly tied to the lattice structure. It is possible that the local rate in non-lattice fixed design case is slower than that in the random design case. For instance, in the extreme case in $d=2$, suppose that the fixed design points are all located on a straight line $\{(x_{1},x_{2})\subset[0,1]^{2}:x_{2}-1/2=\theta(x_{1}-1/2),0\leq x_{1}\leq 1\}$ with $\theta<0$. Then we do not have consistent estimation in general as $g(\cdot)\equiv f(\cdot,\theta\cdot)$ may not be monotonically non-decreasing. The situations for general fixed designs will be more complicated. Although we expect that the local rates in ‘most’ fixed design cases will match that in the random design case, it remains an open question to give a complete characterization.

2.4. An illustrative simulation result

We present here an illustrative simulation result to assess the accuracy of the distributional approximation in Theorem 1 for the case $d=2$ and $\alpha_{1}=\alpha_{2}=1$.

We consider two isotonic regression functions: $f_{1}(x)=e^{x_{1}+x_{2}}$ and $f_{2}(x)=e(x_{1}+x_{2})$, and $x_{0}=(1/2,1/2)$. Clearly, $f_{2}$ is linearization of $f_{1}$ at $x_{0}$. In particular, the product of the partial derivatives of the two isotonic functions are the same at $x_{0}$, i.e., $K(f_{1},x_{0})=K(f_{2},x_{0})$. Theorem 1 indicates that the limiting distributions for the weighted statistics $n^{1/4}(\hat{f}_{n}(x_{0})-f_{i}(x_{0}))$ will be the same for $i=1,2$.

We consider the following lattice sizes in the simulation: $15\times 15,20\times 20,25\times 25,30\times 30$. Figure 3 plots the empirical cumulative distribution functions for the weighted statistics $n^{1/4}(\hat{f}_{n}(x_{0})-f_{i}(x_{0}))(i=1,2)$ based on $B=300$ repetitions with i.i.d. normal errors $\mathcal{N}(0,1)$; it shows that even for the quite small lattice of size $15\times 15$, the max-min block estimator (1.4) already achieves reasonable distributional approximation. Not surprisingly, the shapes of the empirical cumulative distribution functions for $f_{1},f_{2}$ are rather similar. This can be further verified through the QQ-plots for different lattice sizes in Figure 4.

3. Local asymptotic minimax lower bound

We derive in Theorem 1 the precise limiting distribution of the max-min block estimator (1.4) with a local rate of convergence and a limit distribution depending on the unknown smoothness of the regression function at the point of interest. It is natural to wonder if the local rate and the limiting distribution are optimal.

Theorem 2**.**

Suppose Assumptions A-B hold, and the errors $\{\xi_{i}\}$ are i.i.d. $\mathcal{N}(0,\sigma^{2})$. Then with $\kappa_{\ast},n_{\ast}$ defined in Theorem 1, we have

[TABLE]

Furthermore, if all mixed derivatives of $f_{0}$ vanish at $x_{0}$, then

[TABLE]

Here $\lVert\bm{\alpha}\rVert_{\infty}\equiv\max_{\kappa_{\ast}\leq k\leq s}\alpha_{k}$, $\ell_{2}^{2}(f,f_{0})\equiv n^{-1}\sum_{i=1}^{n}(f(X_{i})-f_{0}(X_{i}))^{2}$ for the fixed lattice design case and $\ell_{2}^{2}(f,f_{0})\equiv P(f-f_{0})^{2}$ for the random design case.

Proof.

See Section 7. ∎

Theorem 2 shows that the max-min block estimator (1.4) enjoys a strong oracle property: Both (i) the adaptive local rate of convergence and (ii) the dependence on the constants, whenever explicit, concerning the unknown regression function $f_{0}$ in the limit distribution are optimal in a local asymptotic minimax sense, up to a constant factor depending only on $d,\lVert\bm{\alpha}\rVert_{\infty},P$. Note that here the local minimax lower bound is computed over an $\ell_{2}$-ball with radius of order $\mathcal{O}(n^{-1/2})$.

Remark 4*.*

The dependence of the constant $L_{d,\lVert\bm{\alpha}\rVert_{\infty},P}$ on $P$ in the second claim of Theorem 2 can be further improved if $s=d$ in the random design setting: a slight modification of the proof of Theorem 2 shows that $L_{d,\lVert\bm{\alpha}\rVert_{\infty},P}=(\pi(x_{0}))^{-1/(2+\sum_{k=1}^{d}\alpha_{k}^{-1})}L_{d,\lVert\bm{\alpha}\rVert_{\infty}}$. The limit distribution of max-min block estimator (1.4) achieves the optimal dependence on $\pi(x_{0})$ in this setting, cf. Remark 1.

Remark 5*.*

In a related block-decreasing density estimation problem, [Pav12] establishes a local minimax lower bound in the special case of $\alpha_{1}=\ldots=\alpha_{d}=1$. Their result and Theorem 2 concern different problems but are similar in spirit. Global minimax lower bounds in $L_{1}$ and global risk bounds for histogram-type estimators in the same model are studied in [BD03].

4. Outline of the proofs

4.1. Outline for the proof of Theorem 1

The proof of Theorem 1 is rather involved, so we highlight the main proof ideas here. Recall that $\kappa_{\ast}=1$ in the random design case. Let

[TABLE]

The components of $r_{n}$ indicate the localization rate along each dimension. Now we may re-parametrize the max-min block estimator (1.4) on the scale of the rate vector $r_{n}$. To this end, let $h_{1}^{\ast},h_{2}^{\ast}\in\mathbb{R}_{\geq 0}^{d}$ be such that

[TABLE]

We remind the reader that $\bar{Y}|_{\cdot}$ is the average response over subsets as defined in (1.6). Such a re-parametrization relates the problem to its limit Gaussian version. Note that the first $\kappa_{\ast}-1$ coordinates of $r_{n}$ is [math], but this is no problem: we will show that such $h_{1}^{\ast},h_{2}^{\ast}$ exist with high probability.

For any $c>1$, define the localized max-min block estimator

[TABLE]

where $\gamma_{\ast}>0$ is chosen large enough. The difference of (4.3) compared with (4.2) is that the range of max and min in the global estimator (4.2) is restricted to a compact rectangle away from [math] and $\infty$ in (4.3), so the squared bias and variance in the partial sum process in (4.3) are on the same order. We may therefore expect a non-degenerate limit theory for properly normalized (4.2) by ‘interchanging’ the limit and max-min operations in (4.3) and showing that (4.2) and (4.3) are sufficiently ‘close’ to each other.

Formally, we need the following localization-delocalization result, due to [PR69].

Proposition 2**.**

Suppose that three sequences of random variables $\{W_{n,c}\}$, $\{W_{n}\}$ and $\{W_{c}\}$ satisfy the following conditions:

1. (1)

$\lim_{c\to\infty}\limsup_{n\to\infty}\mathbb{P}(W_{n,c}\neq W_{n})=0$. 2. (2)

For every $c>0$, $W_{n,c}\rightsquigarrow W_{c}$ as $n\to\infty$. 3. (3)

$\lim_{c\to\infty}\mathbb{P}(W_{c}\neq W)=0$.

Then $W_{n}\rightsquigarrow W$ as $n\to\infty$.

We choose

[TABLE]

Now we need to verify the conditions of Proposition 2 for the above defined $W_{n,c}$ and $W_{n}$, and find out the limit process $W_{c}$.

To illustrate the most important ideas in our proof, we focus on the simplest 2-dimensional balanced fixed lattice design case where $\alpha_{1}=\alpha_{2}=1$, $r_{n}=(n^{-1/4},n^{-1/4})$ and $\omega_{n}=n^{-1/4}$ in (4.1), cf. Assumption A.

The first step is to show that the block max-min estimator (4.2) can be localized through its local version (4.3):

[TABLE]

It is easy to see that proving (4.4) reduces to showing that ${h_{1}^{\ast}}$ and ${h_{2}^{\ast}}$ are both bounded away from $\infty$ and [math] in probability, which we refer to as the large deviation and small deviation problems respectively. In other words, neither the bias nor the variance of the partial sum process in (4.2) within the block $[x_{0}-h_{1}^{\ast}r_{n},x_{0}+h_{2}^{\ast}r_{n}]$ will be too large. We accomplish this goal for, e.g. $h_{2}^{\ast}$, in several steps:

(a) First, similarly as in many one-dimensional problems, we establish a local rate of convergence:

[TABLE]

(b) Next we handle the large deviation problem. In other words, we want to show that $\max\{(h_{2}^{\ast})_{1},(h_{2}^{\ast})_{2}\}\leq c$ with high probability for large $c>0$. By the max-min formula,

[TABLE]

The bias term can be handled via (localized) Taylor expansion: suppose $\max\{(h_{2}^{\ast})_{1},(h_{2}^{\ast})_{2}\}>c\gg 1$ (see for example the thick red and blue lines in the left panel of Figure 5), then

[TABLE]

The noise term is essentially contributed by the shaded area in the left panel of Figure 5:

[TABLE]

Combining the above estimates, if $\max\{(h_{2}^{\ast})_{1},(h_{2}^{\ast})_{2}\}>c$, then

[TABLE]

which by (a) should occur with small probability for large $c>0$.

(c) Finally we handle the small deviation problem. By (b), we may assume that $\max\{(h_{2}^{\ast})_{1},(h_{2}^{\ast})_{2}\}\leq c$. Let $0<b<\gamma_{\ast}$ be constants to be determined. Suppose now $(h_{2}^{\ast})_{1}<c^{-\gamma_{\ast}}$ (as in the blue strip in the right panel of Figure 5). Using the max-min formula again (but in a different way) and lower bounding the bias yield that

[TABLE]

The idea here is to choose a larger block for $h_{1}$ compared with the localized block for $h_{2}^{\ast}$. This creates a large positive fluctuation of the noise process $\mathbb{G}(\cdot,\cdot)$ within the shaded region that dominates the relatively small fluctuation within the region in the red dashed line; see the right panel in Figure 5. Indeed, it can be shown that for $\gamma_{\ast}<b+1$, the following small fluctuation holds with high probability: $\mathbb{G}(h_{1},h_{2}^{\ast})-\mathbb{G}(h_{1},0)\geq-C_{2}\cdot\sqrt{c^{2-\gamma_{\ast}}\log c}$ for large $C_{2}>0$. The scaling $c^{2-\gamma_{\ast}}$ is the order of the area of the region within the red dashed line in the right panel of Figure 5. On the other hand, the large positive fluctuation $\max_{0\leq(h_{1})_{1}\leq c^{-b},0\leq(h_{1})_{2}\leq c^{2}}\mathbb{G}(h_{1},0)\geq C_{3}\cdot\sqrt{c^{2-b}}$ holds for small $C_{3}>0$ with high probability (see the shaded area in the right panel of Figure 5). Therefore with high probability, for $c$ large,

[TABLE]

Now by choosing $b>6$ and $\gamma_{\ast}\in(b,b+1)$, the above display can only occur with small probability for large $c$ by (a), so $(h_{2}^{\ast})_{1}>c^{-\gamma_{\ast}}$ with high probability.

Once the first step (4.4) is completed, we may proceed with the second step and conclude that

[TABLE]

To establish the weak convergence in the above display, we need to establish weak convergence of the process $\mathbb{U}_{n}$ to $\mathbb{U}$ in $\ell^{\infty}([c^{-\gamma_{\ast}}\bm{1},c\bm{1}]\times[c^{-\gamma_{\ast}}\bm{1},c\bm{1}])$. Finite dimensional convergence follows immediately by the Taylor expansion in Assumption A. Asymptotic equicontinuity will be verified by general tools developed for uniform central limit theorems for partial sum processes (essentially) in [Ale87] and further developed in [vdVW96]. Note that such asymptotic equicontinuity is possible as the max-min formula searches over rectangles, rather than upper and lower sets as for the least squares estimator.

The last step attempts at localizing the limit of $W\equiv W_{\infty}$ by showing that ${\tilde{h}_{1}},{\tilde{h}_{2}}$ are bounded away from $\infty$ and [math] in probability, where $\tilde{h}_{1},\tilde{h}_{2}$ are such that $W=\mathbb{U}(\tilde{h}_{1},\tilde{h}_{2})$. This problem can be viewed as the limit Gaussian analogue of the first step, and therefore shares a similar proof strategy as detailed above, but further simplifications are possible due to the exact Gaussian structure in the limit.

Finally we list the key properties of the partial sum and limit processes that are used in the proofs. For simplicity, we only consider fixed balanced lattice design with $\alpha_{1}=\ldots=\alpha_{d}=1$. Fix $\rho\in(0,1)$. Let $\omega_{n,\rho}\equiv n^{-\frac{1-\rho}{1+d(1-\rho)}}$, $r_{n,\rho}\equiv\omega_{n,\rho}\bm{1}$ and for any $h_{1},h_{2}\in\mathbb{R}_{\geq 0}^{d}$, let

[TABLE]

Suppose the following properties hold:

1. (P1)

$\mathbb{G}_{n,\rho}\rightsquigarrow\mathbb{G}_{\rho}$ in $\ell^{\infty}([0,c\bm{1}]\times[0,c\bm{1}])$ for any $c>0$, and the limit process $\mathbb{G}_{\rho}(\cdot,\cdot)$ is separable and self-similar with index $\rho\in(0,1)$ in the sense that for any $\gamma\in\mathbb{R}_{\geq 0}^{d}$, \mathbb{G}_{\rho}(\gamma\cdot,\gamma\cdot)=_{d}\big{(}\prod_{k=1}^{d}\gamma_{k}\big{)}^{\rho}\cdot\mathbb{G}_{\rho}(\cdot,\cdot). 2. (P2)

It holds that

[TABLE] 3. (P3)

\mathbb{P}\big{(}\sup_{0\leq h\leq\bm{1}}\mathbb{G}_{\rho}(h,0)\leq 0\big{)}=\mathbb{P}\big{(}\sup_{0\leq h\leq\bm{1}}\mathbb{G}_{\rho}(0,h)\leq 0\big{)}=0.

Then

[TABLE]

where K_{\rho}(f_{0},x_{0})\equiv\big{\{}\prod_{k=1}^{d}\big{(}\partial_{k}f_{0}(x_{0})/2\big{)}\big{\}}^{\frac{1-\rho}{1+d(1-\rho)}} and

[TABLE]

Some comments on the properties (P1)-(P3):

• •

(P1) requires a functional limit theory for the process $\mathbb{G}_{n,\rho}$ to its limit $\mathbb{G}_{\rho}$ over all compact rectangles. The self-similarity index $\rho\in(0,1)$ reflects the dependence structure within the errors $\{\xi_{i}\}$. In the i.i.d. case as considered in this paper, $\rho=1/2$.

• •

(P2) requires some uniform control of the moduli of the processes $\mathbb{G}_{n,\rho},\mathbb{G}_{\rho}$. That the first term in (P2) being stochastically bounded can also be written as

[TABLE]

For i.i.d. errors, this can established by martingale properties of the partial sum process. The stochastic boundedness of the second term in (P2) can be verified by good tail estimates on $\mathbb{G}_{\rho}$.

• •

(P3) is a regularity condition on the limit process $\mathbb{G}_{\rho}$. In the univariate case with i.i.d. errors, this can be easily verified by the reflection principle of Brownian motion.

It is also possible to consider, with substantially increased technicalities, more general assumptions on the local smoothness of $f_{0}$ at $x_{0}$ and the design points as in Assumptions A-B, but we shall omit these details here.

4.2. Outline for the proof of Theorem 2

The basic minimax machinery we use is the following.

Proposition 3**.**

Suppose that the errors $\xi_{i}$’s are i.i.d. $\mathcal{N}(0,\sigma^{2})$. Let $f_{n}$ be such that $n\ell_{2}^{2}(f_{n},f_{0})\leq\alpha$ and $\lvert f_{n}(x_{0})-f_{0}(x_{0})\rvert\geq\gamma_{n}$. Then

[TABLE]

The $\ell_{2}$ metric is defined in the statement of Theorem 2.

Results of this type are well-known in the context of density estimation [GJ95, Jon00], which can also be viewed a special case of minimax reduction scheme with two hypothesis (cf. [Tsy09]). We provide a (short) proof in Section 7 in the context of regression for the convenience of the reader.

Now the problem reduces to that of finding a permissible perturbation $f_{n}$. By the second step in the outline for the proof of Theorem 1, the constants concerning the unknown regression function in the limiting distribution essentially come from the local Taylor expansion, so it is tempting to consider the local perturbation function of the form

[TABLE]

with a good choice of $h\in\mathbb{R}^{d}$. The complication arises from the fact that $\tilde{f}_{n}\notin\mathcal{F}_{d}$ in general, so suitable modifications are needed for a valid construction of $f_{n}$. Details can be found in Section 7.

5. Discussion and final remarks

We developed in this paper pointwise limiting distribution theory for the max-min block estimator (1.4) under local smoothness conditions at a fixed point, and considered both fixed lattice and random designs. One important question is how to use this limit distribution theory for inference. The common bootstrap is known to be inconsistent in a closely related univariate monotone density estimation problem [Kos08, SBW10], so simple bootstrap procedures are unlikely to succeed here as well. It is an important yet non-trivial task to develop a tuning-free inference method for the multiple isotonic regression model (1.3), even with the limiting distribution theory developed in this paper. A detailed study for the inference problem will therefore be pursued elsewhere.

It should however be mentioned that the limit distribution theory is foundational for further developments of inference methods. In the univariate isotonic regression, the limit distribution theory [Bru70, Wri81] is essential both in terms of the results and the proof techniques for the popular inference procedure based on likelihood ratio test developed much later (cf. [BW01, Ban07]). We expect that the results and proof techniques in this paper will also be useful in this and other directions. However, such developments would not be parallel to the univariate analysis based on the linearity of the monotonicity relationship graph.

From another angle, the limiting distribution theory developed in this paper differs markedly from the univariate cases where usually maximum likelihood/least squares estimators are studied. The common difficulty for these MLE/LSEs in higher dimensions is that the underlying ‘empirical process’ (= partial sum process in (1.4) in our setting) is not tight in the large sample limit, so it is hard to obtain limit distribution theories for these estimators. Our results and techniques can therefore also be viewed as a compliment to the extensive literature on the limit theories for univariate MLE/LSEs.

6. Proof of Theorem 1

Let $\varepsilon_{0}>0$ be such that $f_{0}$ is (sufficiently) differentiable on the rectangle $[x_{0}-\varepsilon_{0}\bm{1},x_{0}+\varepsilon_{0}\bm{1}]$. Recall $\kappa_{\ast}=1$ in the random design case, and $\omega_{n}\in\mathbb{R},r_{n}\in\mathbb{R}^{d}$ are defined in (4.1). Let $d_{\ast}\equiv d-\kappa_{\ast}+1$ and $s_{\ast}\equiv s-\kappa_{\ast}+1$. We often omit the requirement that $[x_{u},x_{v}]\cap\{X_{i}\}\neq\emptyset$ in (1.4) for notational simplicity. In the random design setting, we assume that $P$ is the uniform distribution on $[0,1]^{d}$ to avoid unnecessary notational digressions; then the covariance structure of $\mathbb{G}$ is given by the simplified expression (2.4).

In the sequel, we consider separately the cases for $1\leq\kappa_{\ast}<s+1$, and $\kappa_{\ast}=s+1$. In the former case, there is at least one non-trivial term in the Taylor expansion of $f_{0}$ at $x_{0}$, so the problem of limiting distribution is essentially local. In the latter case, since there is only noise present, so the problem is non-local.

6.1. Local rate of convergence

In this subsection we establish the local rate of convergence for $\hat{f}_{n}(x_{0})$. This corresponds to step (a) in Section 4.1.

Proposition 4**.**

Let $1\leq\kappa_{\ast}<s+1$. Assume the same conditions as in Theorem 1. Then $\hat{f}_{n}(x_{0})-f_{0}(x_{0})=\mathcal{O}_{\mathbf{P}}(\omega_{n})$.

We need the following to control the contribution of the noise.

Lemma 2**.**

Assume the same conditions as in Theorem 1. Let $r_{n}$ be as in (4.1). Then for any fixed $\tau>0$, in both fixed lattice and random design cases, we have $\sup_{h>0}\lvert\bar{\xi}|_{[x_{0}-hr_{n},x_{0}+\tau r_{n}]}\rvert=\mathcal{O}_{\mathbf{P}}(\omega_{n})$.

Proof.

See Section 8. ∎

Proof of Proposition 4.

Let $x_{u}^{\ast}$ be such that $\hat{f}_{n}(x_{0})=\min_{x_{v}\geq x_{0}}\bar{Y}|_{[x_{u}^{\ast},x_{v}]}$. Fix a small enough $\tau>0$. Since for $n$ large, $[x_{0},x_{0}+\tau r_{n}]\cap\{X_{i}\}\neq\emptyset$ holds in the fixed lattice design case, and with probability tending to one in the random design case, using the max-min formula we have

[TABLE]

In both fixed lattice and random design cases, by monotonicity of $f_{0}$, we have for $n$ large enough,

[TABLE]

On the other hand, in both fixed lattice and random design cases, Lemma 2 entails that

[TABLE]

The one-sided claim follows by combining (6.1)-(6.3). The other direction follows from similar arguments. ∎

6.2. Localizing the estimator

In this subsection, we tackle the large and small deviation problems, i.e. steps (b)-(c), as described in Section 4.1.

Proposition 5**.**

Let $1\leq\kappa_{\ast}<s+1$. Assume the same conditions as in Theorem 1 and $\kappa_{\ast}$ is unique. Then

[TABLE]

In other words, $\lim_{c\to\infty}\limsup_{n\to\infty}\mathbb{P}(W_{n,c}\neq W_{n})=0.$

Proof.

(Step 1: the large deviation problem). In this step we handle the large deviation problem. For the proof in this step only, define $r_{n}\equiv(\omega_{n}^{1/\alpha_{1}},\ldots,\omega_{n}^{1/\alpha_{d}})$ for notational convenience. Let $h_{1}^{\ast},h_{2}^{\ast}$ be defined as in (4.2) but using the current $r_{n}$, and

[TABLE]

We will show that

[TABLE]

We only show this for \mathbb{P}\big{(}h_{2}^{\ast}\notin H_{c}\big{)}; the situation for $h_{1}^{\ast}$ is similar.

For any $h_{2}^{\ast}$, let $\bar{h}_{2}^{\ast}\equiv h_{2}^{\ast}\wedge(\varepsilon_{0}(r_{n})_{1}^{-1},\ldots,\varepsilon_{0}(r_{n})_{d}^{-1})$. Clearly $\bar{h}_{2}^{\ast}\leq h_{2}^{\ast}$. Then by the max-min formula and the monotonicity of $f_{0}$, in the fixed lattice design case it holds for any $h_{1}\geq 0$ that

[TABLE]

(Step 1a: fixed design, effective dimension). First we will show that we may take $(h_{2}^{\ast})_{\ell}=0$ for $1\leq\ell\leq\kappa_{\ast}-1$ with high probability as $n\to\infty$. Note this is only for the fixed lattice design case. Since we have a lattice design, we only need to show that $(h_{2}^{\ast})_{\ell}<n^{-\beta_{\ell}}(r_{n})_{\ell}^{-1}$ for $1\leq\ell\leq\kappa_{\ast}-1$. On the event $\cup_{1\leq\ell\leq\kappa_{\ast}-1}\{(h_{2}^{\ast})_{\ell}\geq n^{-\beta_{\ell}}(r_{n})_{\ell}^{-1}\}$, we have by Lemma 9 that for $n$ large enough,

[TABLE]

where the last inequality follows from Proposition 1 that $\min_{1\leq\ell\leq\kappa_{\ast}-1}n^{-\beta_{\ell}}(r_{n})_{\ell}^{-1}=n^{\delta}$ for some $\delta>0$. On the other hand, (a slight modification of) Lemma 2 yields that

[TABLE]

Combined with (6.6) using $h_{1}=(1/2)\bm{1}_{[\kappa_{\ast}:d]}$, this shows that on the event $\cup_{1\leq\ell\leq\kappa_{\ast}-1}\{(h_{2}^{\ast})_{\ell}\geq 1\}$,

[TABLE]

which can only occur with arbitrarily small probability according to Proposition 4. This means

[TABLE]

Hence we may take $(h_{2}^{\ast})_{\ell}=0$ for $1\leq\ell\leq\kappa_{\ast}-1$ from now on. We remind again the reader that this claim is for fixed lattice design only.

(Step 1b: fixed and random designs). Next, consider the event $\{\max_{\kappa_{\ast}\leq k\leq d}(h_{2}^{\ast})_{k}>c\}$. For $c>\max_{s+1\leq k\leq d}(1-x_{0})_{k}$, we only need to consider the event $\{\max_{\kappa_{\ast}\leq k\leq s}(h_{2}^{\ast})_{k}>c\}$. Again using Lemma 9, for $c,n$ large enough, and with probability at least $1-\mathcal{O}(n^{-2})$ in the random design case,

[TABLE]

For the noise term, (6.7) holds in both fixed lattice and random design cases. Hence on the intersection of the event $\{\max_{\kappa_{\ast}\leq k\leq d}(h_{2}^{\ast})_{k}>c\}$ and an event with probability tending to $1$,

[TABLE]

holds in both fixed lattice and random design cases. However, in view of Proposition 4, this occurs with arbitrarily small probability for large values of $c>0,n\in\mathbb{N}$. So the event $\{\max_{\kappa_{\ast}\leq k\leq d}(h_{2}^{\ast})_{k}>c\}$ must occur with arbitrarily small probability for $c,n$ large enough. This proves (6.5).

(Step 2: the small deviation problem). In this step we handle the small deviation problem. $r_{n}$ is now defined as in (4.1). Fix $\varepsilon>0$. By Step 1, we may choose $c>0,n\in\mathbb{N}$ large enough such that the event $\Omega_{\varepsilon,c}^{(0)}\equiv\{h_{2}^{\ast}\in H_{c}\}$ holds with probability at least $1-\varepsilon$. Let $a,b,\gamma_{\ast}>0$ with $a>1,0<b<\gamma_{\ast}<b+(a-1)$ be constants to be determined later on, and let $\mathcal{H}_{a,b,\gamma_{\ast}}(c)\equiv\{(h_{1},h_{2})\in\mathbb{R}_{\geq 0}^{d}\times\mathbb{R}_{\geq 0}^{d}:0\leq(h_{1})_{k}\leq c^{a}\bm{1}_{\kappa_{\ast}\leq k\leq s}+(x_{0})_{k}\bm{1}_{s+1\leq k\leq d},0\leq(h_{1})_{d}\leq c^{-b},0\leq(h_{2})_{k}\leq c\bm{1}_{\kappa_{\ast}\leq k\leq s}+(1-x_{0})_{k}\bm{1}_{s+1\leq k\leq d},0\leq(h_{2})_{d}\leq c^{-\gamma_{\ast}}\}$ be defined as in Lemma 14. Consider the event $\Omega^{(1)}_{c}\equiv\{(h_{2}^{\ast})_{d}<c^{-\gamma_{\ast}}\}$. For simplicity of notation, we consider $s<d$; the case $s=d$ follows similarly with slightly different estimates due to Lemma 14. Let $Z_{ni}$ be defined by

[TABLE]

It is verified in the proof of Lemma 3 ahead that for any finite $\tau>0$,

[TABLE]

Hence by Lemma 14, as long as $c>0,n\in\mathbb{N}$ are large enough, there exists a constant $C_{1}=C_{1}(d,\sigma,a)$ such that the event

[TABLE]

holds with probability at least $1-\varepsilon$. On the other hand, by Lemma 16, for $u\equiv\sqrt{c^{as_{\ast}-b}/(x_{0})_{d}}\cdot\rho_{\varepsilon}$ where $\rho_{\varepsilon}$ is taken from Lemma 16, if $a>1$ and $c>1$, it holds for $n$ large enough that

[TABLE]

Hence there exists some constant $C_{2}=C_{2}(x_{0},\rho_{\varepsilon})>0$ such that the event

[TABLE]

holds with probability at least $1-\varepsilon$ for $n$ large enough.

(Step 2a: fixed design, noise). In the fixed lattice design case, on the event $\Omega_{\varepsilon,c}^{(0)}\cap\Omega_{c}^{(1)}\cap\Omega_{\varepsilon}^{(2)}\cap\Omega_{\varepsilon}^{(3)}$, we have for $c>0,n\in\mathbb{N}$ large enough,

[TABLE]

(Step 2b: random design, noise). In the random design case, let $h_{1}(c),h_{2}(c)$ be such that $(h_{1}(c))_{k}=c^{a}\bm{1}_{\kappa_{\ast}\leq k\leq s}+(x_{0})_{k}\bm{1}_{s+1\leq k\leq d-1}+c^{-b}\bm{1}_{k=d}$, and $(h_{2}(c))_{k}=c\bm{1}_{\kappa_{\ast}\leq k\leq s}+(1-x_{0})_{k}\bm{1}_{s+1\leq k\leq d-1}+c^{-\gamma_{\ast}}\bm{1}_{k=d}$. Using Bernstein’s inequality,

[TABLE]

where $\sigma_{c}^{2}\equiv\mathrm{Var}(\bm{1}_{X\in[x_{0}-h_{1}(c)r_{n},x_{0}+h_{2}(c)r_{n}]})\approx c^{as_{\ast}-b}\prod_{k=\kappa_{\ast}}^{d}(r_{n})_{k}$ for $c,n$ large. Hence the event

[TABLE]

occurs with probability at least $1-Ce^{-C^{-1}n\sigma_{c}^{2}}$. So, in the random design setting, on the event $\Omega_{\varepsilon,c}^{(0)}\cap\Omega_{c}^{(1)}\cap\Omega_{\varepsilon}^{(2)}\cap\Omega_{\varepsilon}^{(3)}\cap\Omega_{c}^{(4)}$, it holds that

[TABLE]

(Step 2c: fixed and random designs, bias). On the other hand, in both fixed lattice and random design cases,

[TABLE]

Combining the estimates (6.11), (6.13) and (6.14), we see that for fixed $\varepsilon>0$, if $c>0,n\in\mathbb{N}$ are chosen large enough, on the intersection of the event $\{(h_{2}^{\ast})_{d}<c^{-\gamma_{\ast}}\}$ and an event with probability at least $1-4\varepsilon$, it holds that

[TABLE]

We choose $a\geq 3,b\geq 2(1+a\max_{\kappa_{\ast}\leq k\leq s}\alpha_{k})+as_{\ast}$ and $\gamma_{\ast}=b+1$, so that the above display can only occur with arbitrarily small probability by Proposition 4 for large $c>0,n\in\mathbb{N}$. Hence the event $\{(h_{2}^{\ast})_{d}<c^{-\gamma_{\ast}}\}$, and thereby $\{\min_{\kappa_{\ast}\leq k\leq d}(h_{2}^{\ast})_{k}<c^{-\gamma_{\ast}}\}$, must occur with arbitrarily small probability for large $c>0$. The small deviation for $h_{1}^{\ast}$ can be handled similarly so we omit the details. ∎

6.3. Compact convergence

In this subsection, we establish the weak convergence of the localized process $W_{n,c}$ to the localized limit $W_{c}$.

Proposition 6**.**

Let $1\leq\kappa_{\ast}<s+1$. Assume the same conditions as in Theorem 1. For any $c>1$,

[TABLE]

Here $\sigma$ and $\mathbb{G}$ are specified in Theorem 1.

We need the following functional central limit theorem.

Lemma 3**.**

For any $h_{1},h_{2}>0$, let

[TABLE]

Then for any $c>1$, $\mathbb{G}_{n}\rightsquigarrow\sigma\cdot\mathbb{G}$ in \ell^{\infty}\big{(}[c^{-\gamma_{\ast}}\bm{1},c\bm{1}]\times[c^{-\gamma_{\ast}}\bm{1},c\bm{1}]\big{)}.

Proof.

See Section 8. ∎

Proof of Proposition 6.

Note that in the fixed lattice design case,

[TABLE]

Here the last equality follows from Lemma 9: for $c^{-\gamma_{\ast}}\bm{1}\leq h_{1},h_{2}\leq c\bm{1}$,

[TABLE]

Since the map $\max_{c^{-\gamma_{\ast}}\bm{1}\leq h_{1}\leq c\bm{1}}\min_{c^{-\gamma_{\ast}}\bm{1}\leq h_{2}\leq c\bm{1}}:\mathbb{R}^{[c^{-\gamma_{\ast}}\bm{1},c\bm{1}]\times[c^{-\gamma_{\ast}}\bm{1},c\bm{1}]}\to\mathbb{R}$ is continuous with respect to $\lVert\cdot\rVert_{\infty}$ on $[c^{-\gamma_{\ast}}\bm{1},c\bm{1}]\times[c^{-\gamma_{\ast}}\bm{1},c\bm{1}]$, the claim of the proposition for the fixed lattice design case follows by Lemma 3 and the continuous mapping theorem. The random design case follows from similar arguments by using Lemma 10. ∎

6.4. Localizing the limit

In this subsection, we establish that the limit $W$ can be localized through $W_{c}$ in the sense described in Section 4.1.

Proposition 7**.**

Let $1\leq\kappa_{\ast}<s+1$. For $c>1$, let $W_{c}$ be as in Proposition 6, and $W\equiv W_{\infty}$. Then \lim_{c\to\infty}\mathbb{P}\big{(}W_{c}\neq W\big{)}=0.

To prove Proposition 7, we need the following.

Lemma 4**.**

Let $W$ be defined as in Proposition 7, then $\lVert W\rVert_{\psi_{2}}<\infty$. Here $\lVert\cdot\rVert_{\psi_{2}}$ is the sub-Gaussian Orcliz norm (definition see e.g. [vdVW96]).

Lemma 5**.**

Let $\mathbb{G}$ be defined as in Theorem 1. Then for any $u\geq 1$,

[TABLE]

Here $C_{d}>0$ is a constant depending only on $d$.

Proofs.

See Section 8. ∎

Proof of Proposition 7.

For simplicity of notation we assume $\sigma=1$ without loss of generality and set $a_{\bm{j}}\equiv\partial^{\bm{j}}f_{0}(x_{0})/(\bm{j}+\bm{1})!$. The strategy of the proof largely follows that of Proposition 5, but with some simplifications. Let

[TABLE]

Let $h_{1}^{\ast},h_{2}^{\ast}\in\mathbb{R}^{d}_{\geq 0}$ be such that $W=\mathbb{U}(h_{1}^{\ast},h_{2}^{\ast})$. Since the Gaussian process $\mathbb{G}$ only depends on the last $d_{\ast}$ coordinates of its arguments, we may assume that $(h_{i}^{\ast})_{\ell}=0$ for $1\leq\ell\leq\kappa_{\ast}-1$ and $i=1,2$.

(Step 1). We will first show that

[TABLE]

where $H_{c}^{\ast}$ is defined in (6.4). We only need to prove that $\{\max_{\kappa_{\ast}\leq k\leq s}(h_{2}^{\ast})_{k}\leq c\}$ holds with large probability for $c$ large. Using a similar argument for the inequality as in the proof of Lemma 9, on the event $\{\max_{\kappa_{\ast}\leq k\leq s}(h_{2}^{\ast})_{k}>c\}$,

[TABLE]

The last inequality follows from Lemma 5. On the other hand, by Lemma 4 we know that $W=\mathcal{O}_{\mathbf{P}}(1)$, this means that necessarily $\lim_{c\to\infty}\mathbb{P}(h_{2}^{\ast}\notin H_{c}^{\ast})=0$. Similarly we can show that $\lim_{c\to\infty}\mathbb{P}(h_{1}^{\ast}\notin H_{c}^{\ast})=0$, thereby proving the claim (6.16).

(Step 2). Next we handle the small deviation problem. Using similar arguments as in the proof of Proposition 5, on the intersection of the event $\{(h_{2}^{\ast})_{d}<c^{-\gamma_{\ast}}\}$ and an event with probability at least $1-4\varepsilon$, it holds that

[TABLE]

as $c\to\infty$ by choosing $a\geq 3,b\geq 2(1+a\max_{\kappa_{\ast}\leq k\leq s}\alpha_{k})+as_{\ast}$ and $\gamma_{\ast}=b+1$. The claim follows from Lemma 4. ∎

6.5. Completion of the proof of Theorem 1 for $1\leq\kappa_{\ast}<s+1$

Proof of Theorem 1.

By Proposition 2 combined with Propositions 5-7, it follows that \omega_{n}^{-1}\big{(}\hat{f}_{n}(x_{0})-f_{0}(x_{0})\big{)} converges in distribution to the desired random variable (up to a scaling factor of $\sigma$). Hence we only need to verify the distributional equality in the statement of the theorem, when all mixed derivatives of $f_{0}$ vanish at $x_{0}$ in $J_{\ast}$. To this end, let $\mathbb{U}$ be defined as in the proof of Proposition 7 in (6.15) (with all mixed derivative terms vanishing). Then for $\gamma_{0},\gamma_{1}=\ldots=\gamma_{\kappa_{\ast}-1}=0,\gamma_{\kappa_{\ast}}\ldots,\gamma_{s},\gamma_{s+1}=\ldots=\gamma_{d}=1$ such that

[TABLE]

we have

[TABLE]

This shows that under the choice (6.17),

[TABLE]

Finally we only need to note that solving (6.17) yields that

[TABLE]

This completes the proof. ∎

6.6. Proof of Theorem 1 for $\kappa_{\ast}=s+1$

Proof of Theorem 1.

The strategy of the proof follows the general principle developed for the case $\kappa_{\ast}<s+1$, so we only provide a sketch for the fixed lattice design case.

First, by similar arguments as in the proof of Proposition 4, we can establish a local rate of convergence

[TABLE]

Second, note that $(h_{1}^{\ast})_{k}\leq(x_{0})_{k},(h_{2}^{\ast})_{k}\leq(1-x_{0})_{k}$ for $s+1\leq k\leq d$ so there is no large deviation problem. For the small deviation problem, let $2<b<\gamma_{\ast}$ be some fixed constants, and consider the event $\Omega_{c}^{(1)}\equiv\{(h_{2}^{\ast})_{d}<c^{-\gamma_{\ast}}\}$. We may show, similar to Lemma 14, that there exists some $C_{1}=C_{1}(\sigma,d)$ such that for $c>0,n\in\mathbb{N}$ large enough, the event

[TABLE]

holds with probability at least $1-\varepsilon$ for $n$ large enough. By Lemma 16, there exists some $C_{2}=C_{2}(\varepsilon)>0$ such that the event $\Omega_{\varepsilon}^{(3)}\equiv\{\textrm{for any }0\leq(h_{2})_{k}\leq(1-x_{0})_{k}\bm{1}_{s+1\leq k\leq d},(h_{2})_{d}=0,\textrm{ there exists }0\leq(h_{1})_{k}\leq(x_{0})_{k}\bm{1}_{s+1\leq k\leq d},(h_{1})_{d}\leq c^{-b}\textrm{ such that }\mathbb{G}_{n}(h_{1},h_{2}\bm{1}_{[1:d-1]})\geq C_{2}\sqrt{c^{-b}}\}$ holds with probability $1-\varepsilon$ for $n$ large enough. On the event $\Omega_{c}^{(1)}\cap\Omega_{\varepsilon}^{(2)}\cap\Omega_{\varepsilon}^{(3)}$, we have

[TABLE]

Hence for $c>0,n\in\mathbb{N}$ large enough, on the intersection of $\Omega_{c}^{(1)}$ and an event with probability at least $1-2\varepsilon$,

[TABLE]

where $C_{3}=C_{3}(C_{1},C_{2},\varepsilon)$. However, by (6.18), this cannot occur with high probability for large $c>0$. This concludes the small deviation problem. The rest of the proofs parallels that in the case $\kappa_{\ast}<s+1$ so we omit details. ∎

7. Proof of Theorem 2

We first prove Proposition 3.

Proof of Proposition 3.

The proof is basically contained in [Tsy09]. We provide some details for the convenience of the reader, only in the context of fixed lattice design case. The random design follows from similar arguments. To match the notation used therein, let $s_{n}=\gamma_{n}/2$. Note that

[TABLE]

where the quantity $p_{e,1}$ is defined in page 80 of [Tsy09]. Let $P_{f}^{n}$ denote the probability measure corresponding to the Gaussian regression model $Y_{i}=f(X_{i})+\xi_{i},1\leq i\leq n$. Then the Kullback-Leibler divergence between $P_{f}^{n}$ and $P_{g}^{n}$ is given by the rescaled discrete $\ell_{2}$ distance between $f$ and $g$:

[TABLE]

Now Theorem 2.2 of [Tsy09] applies to conclude. ∎

Proof of Theorem 2.

We assume that $f_{0}$ is locally $C^{\max_{1\leq k\leq s}\alpha_{k}}$ at $x_{0}$ with vanishing mixed derivatives for simplicity of notation. The lower bound for the case $\kappa_{\ast}=s+1$ is trivial, so we only consider the case $\kappa_{\ast}<s+1$. Consider the fixed lattice design case. Let $r_{n}$ be defined as in (4.1), and $h$ be a vector defined by

[TABLE]

where

[TABLE]

and $\tau>0$ is a small enough fixed constant. Consider local perturbation functions

[TABLE]

Clearly $f_{n}\in\mathcal{F}_{d}$ by construction. Let $\mathcal{X}_{n}\equiv\{x\in\{X_{i}\}:f_{n}(x)\neq f_{0}(x)\}$. We claim that for $n$ large enough,

[TABLE]

To prove (7.1), it suffices to show that for any $x\in\{X_{i}\}$, $x_{0}-\varepsilon_{0}\bm{1}\leq x\leq x_{0},x\ngeq x_{0}-2d\lVert\bm{\alpha}\rVert_{\infty}\cdot hr_{n}$, we have $f_{0}(x)<f_{0}(x_{0}-hr_{n})$. To see this, first note that

[TABLE]

On the other hand,

[TABLE]

Here $\delta>0$ by Proposition 1. Comparing the above two displays proves the claim (7.1). Therefore, with

[TABLE]

we have for $n$ large enough, in the fixed lattice design case,

[TABLE]

In the random design case, we can deduce the same inequality as above up to a constant factor depending on $P$. On the other hand, it follows from direct Taylor expansion that

[TABLE]

Now apply Proposition 3 to conclude the lower bound when $\kappa_{\ast}<s+1$.

Use techniques similar to the proof of the inequality of Lemma 9, we may control the cross terms with mixed derivatives in the Taylor expansion when such terms do not vanish. ∎

8. Remaining proofs

8.1. Proof of Lemma 1

Proof of Lemma 1.

(1) If $\alpha_{k}=1$, then it is easy to see by monotonicity that the assumption requires $\partial_{k}^{\alpha_{k}}f_{0}(x_{0})>0$. If $\alpha_{k}>1$, since for any $x\in[0,1]^{d}$ close enough to $x_{0}$ with $x_{j}=(x_{0})_{j},j\neq k$,

[TABLE]

it follows that $\alpha_{k}$ must be odd and $\partial_{k}^{\alpha_{k}}f_{0}(x_{0})>0$.

(2) Let $J_{0}\equiv\{\bm{j}\in J:0<\sum_{k=1}^{s}j_{k}/\alpha_{k}<1\}$. Suppose $\partial^{\bm{j}}f_{0}(x_{0})\neq 0$ for some $\bm{j}\in J_{0}$. We will prove that $\partial_{k}^{\alpha_{k}^{\prime}}f_{0}(x_{0})\neq 0$ for some $1\leq\alpha_{k}^{\prime}<\alpha_{k}$ and $1\leq k\leq s$. Let $\omega_{n}\searrow 0$ and $r_{n}\equiv(\omega_{n}^{1/\alpha_{1}},\ldots,\omega_{n}^{1/\alpha_{d}})$, and $D_{n}\equiv\max_{\bm{0}\neq\bm{j}\in J,\partial^{\bm{j}}f_{0}(x_{0})\neq 0}r_{n}^{\bm{j}}/\omega_{n}$. Since $r_{n}^{\bm{j}}/\omega_{n}=\omega_{n}^{\sum_{k=1}^{s}j_{k}/\alpha_{k}-1}$, we have $D_{n}\to\infty$. Let $t=(x-x_{0})/r_{n}\in\mathbb{R}^{d}$ and

[TABLE]

Then $Q$ is a non-decreasing polynomial in the sense that if $t_{1}\leq t_{2}$, we have $Q(t_{1})\leq Q(t_{2})$. Note that in the above display the summation is over $J_{0}$ instead of $J$ since we assumed that $\partial^{\bm{j}}f_{0}(x_{0})\neq 0$. Since $Q$ only depends on its first $s$ arguments, we slightly abuse the notation for $Q$ as a polynomial of $(t_{1},\ldots,t_{s})$ in the sequel. As $\partial_{s}^{\alpha_{s}^{\prime}}f_{0}(x_{0})=0$ for all $1\leq\alpha_{s}^{\prime}<\alpha_{s}$, we have $Q(0,\ldots,0,t_{s})=0$. Assume now $Q(0,\ldots,0,t_{k+1},\ldots,t_{s})=0$ while $Q(0,\ldots,0,t_{k},\ldots,t_{s})\neq 0$. We may write

[TABLE]

with some $\alpha_{k}^{\prime}\geq 1$ and $Q_{\alpha_{k}^{\prime}}(t_{k+1},\ldots,t_{s})\neq 0$. By monotonicity of $Q$, $\alpha_{k}^{\prime}$ is odd. Furthermore, for $t_{k}=\pm 1$ and any $k+1\leq\ell\leq s$, we have

[TABLE]

Hence $\partial_{k}^{\alpha_{k}^{\prime}}f_{0}(x_{0})/(\alpha_{k}^{\prime})!=Q_{\alpha_{k}^{\prime}}(0)=Q_{\alpha_{k}^{\prime}}(t_{k+1},\ldots,t_{s})\neq 0$.

The boundedness of $\omega_{n}^{-1}\lvert f_{0}(x)-f_{0}(x_{0})\rvert$ for $x\in[0,1]^{d}$ such that $\lvert(x-x_{0})_{k}\rvert\leq(r_{n})_{k},1\leq k\leq d$, follows since for $\bm{j}\in J_{\ast}$, $\omega_{n}^{-1}\lvert(x-x_{0})^{\bm{j}}\rvert\leq\omega_{n}^{-1}r_{n}^{\bm{j}}=1$.

(3) Suppose $\mathrm{gcd}(\alpha_{k_{1}},\alpha_{k_{2}})=1$ for all $1\leq k_{1}<k_{2}\leq s$. Then for any $\bm{j}$, $\sum_{k=1}^{d}j_{k}/\alpha_{k}=1$ and $j_{k}>0$, we have $\alpha_{k}|j_{k}$ and $j_{\ell}=0$ for all $\ell\neq k$. This means $J_{1}=\emptyset$. On the other hand, if $\mathrm{gcd}(\alpha_{1},\alpha_{2})>1$, write $\alpha_{1}=m_{1}m,\alpha_{2}=m_{2}m$ for some positive odd numbers $m_{1},m_{2},m$ with $m\geq 3$. Then with $j_{1}\equiv m_{1}p,j_{2}\equiv m_{2}q$ where $p,q\geq 1,p+q=m$ (such $p,q$ exist as $m\geq 3$), we have $j_{1}/\alpha_{1}+j_{2}/\alpha_{2}=1$, so $J_{1}\neq\emptyset$.

(4) Without loss of generality suppose $\alpha_{1}=m_{1}m,\alpha_{2}=m_{2}m$ for some positive odd numbers $m_{1},m_{2},m$ with $m\geq 3$ and $d=2$. Consider

[TABLE]

Let $y_{i}\equiv t_{i}^{m_{i}},i=1,2$ and $g(y)\equiv y_{1}^{m}+y_{1}^{m-1}y_{2}+y_{1}y_{2}^{m-1}+y_{2}^{m}$. Then $\partial_{1}g(y)=my_{1}^{m-1}+(m-1)y_{1}^{m-2}y_{2}+y_{2}^{m-1}\geq 0$, as $\lvert y_{1}^{m-2}y_{2}\rvert\leq\frac{m-2}{m-1}y_{1}^{m-1}+\frac{1}{m-1}y_{2}^{m-1}$. So with $x\equiv x_{0}+t$, $\partial_{1}f(x)=(\partial/\partial t_{1})f(x_{0}+t)=(\partial g/\partial y_{1})(\partial y_{1}/\partial t_{1})\geq 0$. Similarly $\partial_{2}f(x)\geq 0$, so $f\in\mathcal{F}_{2}$. It is easy to check that $f$ satisfies Assumption A with $\bm{\alpha}=(\alpha_{1},\alpha_{2})=(m_{1}m,m_{2}m)$, but $\partial^{m_{1}(m-1),m_{2}}f(x_{0}),\partial^{m_{1},m_{2}(m-1)}f(x_{0})\neq 0$. ∎

8.2. Proof of Lemma 2

Proof of Lemma 2.

Without loss of generality, we assume $\tau=1$. Let $x_{n}\equiv x_{0}+\tau r_{n}$. We only prove the lemma for the random design case; the fixed lattice design case follows from simpler arguments. Let $s=d$ for simplicity. Let $\mathcal{E}_{n}$ be the event specified by Lemma 10 with $g\equiv 1$ therein. Then,

[TABLE]

as desired. ∎

8.3. Proof of Lemma 3

We need the following to prove Lemma 3.

Lemma 6** ([Ale87]; see also Theorem 2.11.9 of [vdVW96]).**

For each $n$, let $Z_{n1},\ldots,Z_{nm_{n}}$ be independent stochastic processes indexed by a totally bounded semi-metric space $(\mathcal{F},\rho)$. Suppose that

1. (1)

$\sum_{i=1}^{m_{n}}\mathbb{E}\lVert Z_{ni}\rVert_{\mathcal{F}}^{2}\bm{1}_{\lVert Z_{ni}\rVert_{\mathcal{F}}>\eta}\to 0$* for every $\eta>0$.* 2. (2)

\sup_{\rho(f,g)<\delta_{n}}\sum_{i=1}^{m_{n}}\mathbb{E}\big{(}Z_{ni}(f)-Z_{ni}(g)\big{)}^{2}\to 0* for every $\delta_{n}\to 0$.* 3. (3)

$\int_{0}^{\delta_{n}}\sqrt{\log\mathcal{N}_{[\,]}(\varepsilon,\mathcal{F},L_{2}^{n})}\ \mathrm{d}\varepsilon\to 0$* for every $\delta_{n}\to 0$. Here the bracketing number $\mathcal{N}_{[\,]}(\varepsilon,\mathcal{F},L_{2}^{n})$ is defined as the minimal number of sets $N_{\varepsilon}$ in a partition $\mathcal{F}\equiv\cup_{j=1}^{N_{\varepsilon}}\mathcal{F}_{\varepsilon j}^{n}$ of the index set into sets $\mathcal{F}_{\varepsilon j}^{n}$ such that for every partitioning set $\mathcal{F}_{\varepsilon j}^{n}$,*

[TABLE]

Then the sequence $\sum_{i=1}^{m_{n}}(Z_{ni}-\mathbb{E}Z_{ni})$ is asymptotically $\rho$-equicontinuous.

Proof of Lemma 3.

We will prove a slightly stronger statement by strengthening the convergence in \ell^{\infty}\big{(}[c^{-\gamma_{\ast}}\bm{1},c\bm{1}]\times[c^{-\gamma_{\ast}}\bm{1},c\bm{1}]\big{)} to convergence in \ell^{\infty}\big{(}[0,c\bm{1}]\times[0,c\bm{1}]\big{)}. We first verify the finite-dimensional convergence. This is the easy step. For fixed lattice design, for any $(h_{1},h_{2}),(h_{1}^{\prime},h_{2}^{\prime})\in\mathbb{R}_{\geq 0}^{d}\times\mathbb{R}_{\geq 0}^{d}$, we only need to note that

[TABLE]

For random design, we may proceed with the above lines conditionally.

Next we verify the asymptotic equicontinuity by Lemma 6. Let

[TABLE]

Let $m_{n}\equiv n$, $\mathcal{F}\equiv\{(h_{1},h_{2})\in[0,c\bm{1}]\times[0,c\bm{1}]\subset\mathbb{R}^{d}_{\geq 0}\times\mathbb{R}^{d}_{\geq 0}\}$. Let $\rho$ be defined as follows: for any $(h_{1},h_{2}),(h_{1}^{\prime},h_{2}^{\prime})\in[0,c\bm{1}]\times[0,c\bm{1}]$,

[TABLE]

where $\lVert\cdot\rVert$ is the standard Euclidean norm, and $\pi_{\kappa_{\ast}}:\mathbb{R}^{d}\to\mathbb{R}^{d_{\ast}}$ is the canonical projection onto last $d_{\ast}$ coordinates. We now verify conditions (1)-(3) of Lemma 6 as follows.

For condition (1), note that $\lVert Z_{ni}\rVert_{\mathcal{F}}=\omega_{n}\lvert\xi_{i}\rvert\bm{1}_{i:x_{0}-cr_{n}\leq X_{i}\leq x_{0}+cr_{n}}$. Hence for any $\eta>0$, in the fixed lattice design case,

[TABLE]

as $n\to\infty$. The random design case follows from obvious modifications using conditioning arguments and the fact that the density $\pi$ is bounded away from $\infty$.

For condition (2), for any $(h_{1},h_{2}),(h_{1}^{\prime},h_{2}^{\prime})$ such that \rho\big{(}(h_{1},h_{2}),(h_{1}^{\prime},h_{2}^{\prime})\big{)}<\delta_{n}, in the fixed lattice design case

[TABLE]

The last inequality follows by Lemma 15. The random design case follows similarly. This verifies (2).

For (3), we can identify $(h_{1},h_{2})\in\mathcal{F}$ as the indicator $\bm{1}_{[x_{0}-h_{1}r_{n},x_{0}+h_{2}r_{n}]}\equiv\tau(h_{1},h_{2})$, where $h_{i}\in[0,c\bm{1}](i=1,2)$. For each $\varepsilon>0$, let h_{\varepsilon}\equiv\varepsilon^{2}/\big{(}\mathbb{E}\lvert\xi_{1}\rvert^{2}d_{\ast}(2c)^{d_{\ast}}2\big{)}. Then

[TABLE]

where $N_{\varepsilon}\leq(c/h_{\varepsilon})^{2d}$.

Now we show that this is a valid $\varepsilon$-bracketing in the sense defined in Lemma 6. For $j=1,\ldots,N_{\varepsilon}$, $\mathcal{F}_{\varepsilon j}^{n}=\{\tau(h_{1}^{\prime},h_{2}^{\prime}):h_{i}\leq h_{i}^{\prime}\leq h_{i}+h_{\varepsilon},i=1,2\}$ for some $h_{1},h_{2}\in\{0,h_{\varepsilon},\ldots,c\}^{d}$. Then, in the fixed lattice design case,

[TABLE]

The random design case follows similarly (with a slight modification of the definition of $h_{\varepsilon}$). On the other hand,

[TABLE]

as $\delta_{n}\to 0$. This verifies (3).

Now we may apply Lemma 6 to conclude the claim of the lemma. ∎

8.4. Proofs of Lemmas 4 and 5

We need Dudley’s entropy integral bound for sub-Gaussian processes, recorded below for the convenience of the reader.

Lemma 7** (Theorem 2.3.7 of [GN15]).**

Let $(T,d)$ be a pseudo metric space, and $(X_{t})_{t\in T}$ be a sub-Gaussian process such that $X_{t_{0}}=0$ for some $t_{0}\in T$. Then

[TABLE]

Here $C>0$ is a universal constant.

The following Gaussian concentration inequality will also be useful.

Lemma 8** (Borell’s Gaussian concentration inequality).**

Let $(T,d)$ be a pseudo metric space, and $(X_{t})_{t\in T}$ be a mean-zero Gaussian process. Then with $\sigma^{2}\equiv\sup_{t\in T}\mathrm{Var}(X_{t})$, for any $u>0$,

[TABLE]

Proof of Lemma 5.

Without loss of generality we assume $s=d$. Let $G$ be the Gaussian process defined on $\mathbb{R}_{\geq 0}^{d}$ through $G(h)\equiv\mathbb{G}(h,\bm{1})$, whose natural induced metric on $\mathbb{R}_{\geq 0}^{d}$ is given by d_{G}^{2}(h_{1},h_{2})\equiv\mathbb{E}(G(h_{1})-G(h_{2}))^{2}=\lambda_{d_{\ast}}\big{(}[-\bm{1}_{[\kappa_{\ast}:d]},h_{1}\bm{1}_{[\kappa_{\ast}:d]}]\Delta[-\bm{1}_{[\kappa_{\ast}:d]},h_{2}\bm{1}_{[\kappa_{\ast}:d]}]\big{)}, where $\lambda_{d_{\ast}}$ is the Lebesgue measure on $\mathbb{R}^{d_{\ast}}$. Then for $u\geq 0$,

[TABLE]

Since for $h_{i}$ such that $0\leq(h_{i})_{k}\leq 2^{\ell_{k}}$, $i=1,2$, Lemma 15 entails that $d_{G}^{2}(h_{1},h_{2})\leq c_{d_{\ast}}\prod_{k=\kappa_{\ast}}^{d}2^{\ell_{k}}\cdot\lVert h_{1}-h_{2}\rVert$, it follows by Dudley’s entropy integral (cf. Lemma 7) that

[TABLE]

where $L(x)=\sqrt{x\log x}$. This implies that for $u$ larger than a constant only depending on $d_{\ast}$,

[TABLE]

where in the last line we used Gaussian concentration inequality (cf. Lemma 8) and the fact that $\sup_{0\leq h_{k}\leq 2^{\ell_{k}},\kappa_{\ast}\leq k\leq d}\mathrm{Var}(G(h))\leq 2^{d_{\ast}}\prod_{k=\kappa_{\ast}}^{d}2^{\ell_{k}}$. Combining (8.1) and (8.2) we see that for $u>0$ large enough,

[TABLE]

This completes the proof. ∎

Proof of Lemma 4.

For simplicity of notation we assume that $\sigma=1$. Then

[TABLE]

Now by Lemma 5 and (8.3), we see that $\lVert(W)_{+}\rVert_{\psi_{2}}<\infty$. Similarly we can establish $\lVert(W)_{-}\rVert_{\psi_{2}}<\infty$. ∎

9. Auxiliary lemmas

Lemma 9**.**

Suppose Assumption A holds. For $\omega_{n}\searrow 0$, let $r_{n}\equiv(\omega_{n}^{1/\alpha_{1}},\ldots,\omega_{n}^{1/\alpha_{d}})$.

• •

(Fixed design) For $h_{1},h_{2}\geq 0$ such that $\max\limits_{i=1,2}\max\limits_{1\leq k\leq d}(h_{i})_{k}(r_{n})_{k}\leq\varepsilon_{0}$,

[TABLE]

Here $A_{q}(t_{1},t_{2})\equiv\sum_{k=-\left\lfloor t_{1}\right\rfloor}^{\left\lfloor t_{2}\right\rfloor}k^{q}/(\left\lfloor t_{1}\right\rfloor+\left\lfloor t_{2}\right\rfloor+1)$ for $t_{1},t_{2}\geq 0$, and \tilde{\omega}_{n}\equiv\omega_{n}\cdot\max\limits_{1\leq k\leq s}\big{(}1\vee(h_{1})_{k}^{\alpha_{k}}\vee(h_{2})_{k}^{\alpha_{k}}\big{)}\equiv\omega_{n}\cdot\big{(}1\vee(h_{1})_{k_{0}}^{\alpha_{k_{0}}}\vee(h_{2})_{k_{0}}^{\alpha_{k_{0}}}\big{)}. Moreover, for $c>0$ large enough, if (h_{2})_{k_{0}}\geq\max\{c,c\max_{1\leq k\leq s}\big{(}1\vee(h_{1})_{k}),n^{-\beta_{k_{0}}}(r_{n})_{k_{0}}^{-1}\}, then

[TABLE]

• •

(Random design) Suppose that $\omega_{n}^{1+\sum_{k}\alpha_{k}^{-1}}\gg\log n/n$, and that $P$ is the uniform distribution on $[0,1]^{d}$. Then for any $c_{0}\geq 1$, uniformly for $h_{1},h_{2}\geq 0$ such that $\max\limits_{i=1,2}\max\limits_{1\leq k\leq d}(h_{i})_{k}(r_{n})_{k}\leq\varepsilon_{0}$ and that $c_{0}^{-1}\bm{1}\leq h_{1}\leq c_{0}\bm{1}$, it holds with probability at least $1-\mathcal{O}(n^{-2})$ that,

[TABLE]

Here $I_{q}(t_{1},t_{2})\equiv\int_{-t_{1}}^{t_{2}}x^{q}\ \mathrm{d}x/(t_{1}+t_{2})$ for $t_{1},t_{2}\geq 0$, and \tilde{\omega}_{n}\equiv\omega_{n}\cdot\max_{1\leq k\leq s}\big{(}1\vee(h_{2})_{k}^{\alpha_{k}}\big{)}\equiv\omega_{n}\cdot\big{(}1\vee(h_{2})_{k_{0}}^{\alpha_{k_{0}}}\big{)}. Moreover, for $c>0$ large enough (possibly depending on $c_{0}$), if $(h_{2})_{k_{0}}\geq c$, then

[TABLE]

In both cases, $C_{\bm{\alpha}}>0$ is a constant only depending on $\bm{\alpha}$.

Proof.

First consider fixed lattice design case. By Assumption A,

[TABLE]

The whole term above is of order $\mathcal{O}(\tilde{\omega}_{n})$ since

[TABLE]

where the last inequality follows from the fact that for any $\bm{j}\in J_{\ast}$, and $h\geq 0$, $h^{\bm{j}}\leq\sum_{k}(j_{k}/\alpha_{k})h_{k}^{\alpha_{k}}$. For the inequality in the statement of the lemma, by monotonicity of $\bar{f_{0}}|_{[x_{0}-h_{1}r_{n},x_{0}+h_{2}r_{n}]}-f_{0}(x_{0})$, we may assume $(h_{2})_{k}=0$ for $k\neq k_{0}$. Then for any $\bm{j}\in J_{\ast}$, if $(h_{2})_{k_{0}}>c>1$,

[TABLE]

For $j_{k_{0}}=\alpha_{k_{0}}$, if $(h_{2}r_{n})_{k_{0}}n^{\beta_{k_{0}}}\geq 1,(h_{2})_{k_{0}}>c\max_{1\leq k\leq s}(h_{1})_{k}$ for a large enough constant $c>0$, using $A_{q}(t_{1},t_{2})\geq\int_{0}^{t_{2}}x^{q}\ \mathrm{d}x/(t_{1}+t_{2}+1)-(t_{1}+1)^{q}$ for $t_{2}\geq 1,t_{1}\geq 0$ and odd positive integer $q$, we have

[TABLE]

Collecting the bounds we have proved the inequality.

Next for random design case, using Lemma 13, with the desired probability we have

[TABLE]

The inequality in the random design case can be proved similarly, but without the constraint that $(h_{2}r_{n})_{k_{0}}n^{\beta_{k_{0}}}\geq 1$ since $I_{q}(t_{1},t_{2})\geq\int_{0}^{t_{2}}x^{q}\ \mathrm{d}x/(t_{1}+t_{2})-t_{1}^{q}$ holds for all $t_{1},t_{2}\geq 0$ and odd positive integer $q$. ∎

Lemma 10**.**

Let $P$ be a probability distribution on $[0,1]^{d}$ with Lebesgue density bounded away from [math] and $\infty$, and let $g:[0,1]^{d}\to\mathbb{R}_{\geq 0}$ be a measurable function that is uniformly bounded by $1$. Then for any $\Gamma\subset[0,1]^{d}$, $L_{n}\geq 1$ and $n$ large enough, it holds with probability at least $1-n^{-2}$ that

[TABLE]

Here $C>0$ is an absolute constant, and $\mathbb{P}_{n}$ is the empirical measure.

We need Talagrand’s concentration inequality [Tal96] for the empirical process in the form given by Bousquet [Bou03], recorded as follows.

Lemma 11** (Theorem 3.3.9 of [GN15]).**

Let $\mathcal{F}$ be a countable class of real-valued measurable functions such that $\sup_{f\in\mathcal{F}}\lVert f\rVert_{\infty}\leq b$. Then

[TABLE]

where $\bar{\sigma}^{2}\equiv\sigma^{2}+2bn^{-1/2}\mathbb{E}\sup_{f\in\mathcal{F}}\lvert\nu_{n}f\rvert$ with $\sigma^{2}\equiv\sup_{f\in\mathcal{F}}\mathrm{Var}_{P}f$, and $\nu_{n}\equiv\sqrt{n}(\mathbb{P}_{n}-P)$.

Let $J(\delta,\mathcal{F},L_{2})\equiv\int_{0}^{\delta}\sup_{Q}\sqrt{1+\log\mathcal{N}(\varepsilon\lVert F\rVert_{Q,2},\mathcal{F},L_{2}(Q))}\ \mathrm{d}\varepsilon$, where the supremum is taken over all finitely discrete probability measures. The following local maximal inequality for the empirical process due to [GK06, vdVW11] is useful.

Lemma 12**.**

Let $\mathcal{F}$ be a countable class of real-valued measurable functions such that $\sup_{f\in\mathcal{F}}\lVert f\rVert_{\infty}\leq 1$, and $X_{1},\ldots,X_{n}$’s are i.i.d. random variables with law $P$. Then with $\mathcal{F}(\delta)\equiv\{f\in\mathcal{F}:Pf^{2}<\delta^{2}\}$,

[TABLE]

Proof of Lemma 10.

Consider the class $\mathcal{F}\equiv\{g\bm{1}_{[h_{1},h_{2}]}:0\leq h_{1}\leq h_{2}\leq 1\}$. Let $\alpha_{n}=\sqrt{L_{n}\log n/n}$ where $L_{n}\geq 1$. For $j\geq 1$, let $\mathcal{F}_{j}\equiv\{g\bm{1}_{[h_{1},h_{2}]}:2^{j-1}\alpha_{n}\leq\sqrt{Pg\bm{1}_{[h_{1},h_{2}]}}\leq 2^{j}\alpha_{n}\}$. Let $\ell$ be the smallest integer such that $2^{\ell}\alpha_{n}\geq 1$. Then by Talagrand’s concentration inequality (cf. Lemma 11) for bounded empirical processes, it follows that for any $s_{j}\geq 0$,

[TABLE]

Hence by a union bound, using the estimate (by e.g. Lemma 12) that $\mathbb{E}\sup_{f\in\mathcal{F}_{j}}\lvert\nu_{n}(f)\rvert\lesssim 2^{j}\alpha_{n}\sqrt{\log n}$, and choosing $s_{j}=3\log n$, it follows that with probability at least $1-n^{-2}$, it holds that

[TABLE]

The proof is complete. ∎

Lemma 13**.**

Let $\omega_{n}\searrow 0$ be such that $\omega_{n}^{1+\sum_{k}\alpha_{k}^{-1}}\gg\log n/n$, and let $r_{n}\equiv(\omega_{n}^{1/\alpha_{1}},\ldots,\omega_{n}^{1/\alpha_{d}})$. Let $P$ be a probability measure on $[0,1]^{d}$ with Lebesgue density bounded away from [math] and $\infty$ and $x_{0}\in(0,1)^{d}$. For any $c_{0}\geq 1$ and $\bm{j}\in J_{\ast}$, we may find some $u_{n}\searrow 0$ such that with probability at least $1-\mathcal{O}(n^{-2})$,

[TABLE]

Here \tilde{\omega}_{n}(h_{1},h_{2};\bm{j})\equiv\omega_{n}\cdot\prod_{k}\big{(}(h_{1})_{k}^{j_{k}}\vee(h_{2})_{k}^{j_{k}}\big{)}, and $L_{n}$ is a slowly growing sequence taken from Lemma 10.

Proof.

For notational simplicity, we prove the case $c_{0}=1$ and write $h_{2}\equiv h$, $\tilde{\omega}_{n}(h)\equiv\tilde{\omega}_{n}(\bm{1},h;\bm{j})$. The general case follows from minor modifications. Let $g_{0}(x)\equiv(x-x_{0})^{\bm{j}}$. By Lemma 10, we only need to prove with the prescribed probability,

[TABLE]

and

[TABLE]

For (9.2), note that for any $h\geq 0$,

[TABLE]

so we may apply Lemma 10 to bound $(\ref{ineq:ratio_random_replace_1})$ by

[TABLE]

Next we consider (9.3). By Lemma 12, since for any $\{\ell_{k}\in\mathbb{Z}_{\geq 0}\}_{k=1}^{d}$, \sup_{2^{\ell_{k}}-1\leq h_{k}\leq 2^{\ell_{k}+1}-1}\mathrm{Var}_{P}\big{(}g_{0}\bm{1}_{[x_{0}-r_{n},x_{0}+hr_{n}]}\big{)}\lesssim r_{n}^{2\bm{j}+\bm{1}}\prod_{k=1}^{d}2^{2\ell_{k}}, we have

[TABLE]

Then with

[TABLE]

for a large enough constant $K>0$, by Talagrand’s concentration inequality (cf. Lemma 11), for $n$ large,

[TABLE]

as desired. ∎

Lemma 14**.**

Let $(a,b,\gamma_{\ast})\in\mathbb{R}_{\geq 0}^{3}$ be such that $a>1,0<b<\gamma_{\ast}<b+(a-1)$. Let $\mathcal{H}_{a,b,\gamma_{\ast}}(c)\equiv\{(h_{1},h_{2})\in\mathbb{R}_{\geq 0}^{d}\times\mathbb{R}_{\geq 0}^{d}:0\leq(h_{1})_{k}\leq c^{a}\bm{1}_{\kappa_{\ast}\leq k\leq s}+(x_{0})_{k}\bm{1}_{s+1\leq k\leq d},0\leq(h_{1})_{d}\leq c^{-b},0\leq(h_{2})_{k}\leq c\bm{1}_{\kappa_{\ast}\leq k\leq s}+(1-x_{0})_{k}\bm{1}_{s+1\leq k\leq d},0\leq(h_{2})_{d}\leq c^{-\gamma_{\ast}}\}$. Let $\mathbb{G}$ be defined as in Theorem 1. Then there exists some constant $C_{d,a}>0$ such that for any $c>1$,

[TABLE]

Proof.

Without loss of generality, we assume $\sigma=1$. Let $\mathbb{W}(h_{1},h_{2})\equiv\mathbb{G}(h_{1},h_{2})-\mathbb{G}(h_{1},h_{2}\bm{1}_{[s+1:d-1]})$. Then the process $\{\mathbb{W}(h_{1},h_{2}):h_{1},h_{2}\geq 0\}$ is a Gaussian process with the natural induced metric $d_{\mathbb{W}}$ given as follows: for $(h_{1},h_{2}),(h_{1}^{\prime},h_{2}^{\prime})$,

[TABLE]

For $(I)$, note that

[TABLE]

The last line follows from Lemma 15. A similar estimate can be obtained for $(II)$. Hence,

[TABLE]

On the other hand, the diameter of the indexing set in the supremum of the process in question can be controlled as follows: If $s<d$,

[TABLE]

If $s=d$, similarly we have $D_{c}^{2}\leq C_{d}\cdot c^{a(s_{\ast}-1)-\gamma_{\ast}}$. Now Dudley’s entropy integral (cf. Lemma 7) yields that

[TABLE]

as desired. ∎

Lemma 15**.**

For any $c_{k}\geq 1,k=\kappa_{\ast},\ldots,d$, let $g:\prod_{k=\kappa_{\ast}}^{d}[0,c_{k}]\times\prod_{k=\kappa_{\ast}}^{d}[0,c_{k}]\to\mathbb{R}$ be defined as $g(h_{1},h_{2})\equiv\prod_{k=\kappa_{\ast}}^{d}(h_{1}+h_{2})_{k}$. Then

[TABLE]

Here $\lVert\cdot\rVert$ is the canonical Euclidean norm on $\mathbb{R}^{d_{\ast}}$.

Proof.

Note that the first order partial derivatives of $g(\cdot,\cdot)$ are all bounded by $\prod_{k=\kappa_{\ast}}^{d}(2c_{k})$, using Taylor expansion yields that

[TABLE]

completing the proof. ∎

Lemma 16**.**

Let $\mathbb{G}$ be defined in Theorem 1. Then

[TABLE]

In particular, for any $\varepsilon>0$, there exists some $\rho_{\varepsilon}>0$ such that

[TABLE]

Proof.

Without loss of generality, we may assume that $\kappa_{\ast}=s+1$, since otherwise we may consider the weaker statement:

[TABLE]

If $\kappa_{\ast}=d$, the claim follows from the standard reflection principle for Brownian motion, so we work with $\kappa_{\ast}\leq d-1$. Let $\mathcal{I}\equiv\{S_{0},S_{1},\ldots,S_{J}\}$, where $0=S_{0}<S_{1}<\ldots<S_{J}=1$ will be determined later on. For $\mathscr{I}\subset\{\kappa_{\ast},\ldots,d\}$, let $\mathbb{W}_{\mathscr{I}}$ be the standard white noise living on the coordinates $\mathscr{I}$, and write $\mathbb{W}\equiv\mathbb{W}_{\{\kappa_{\ast},\ldots,d\}}$ for notational simplicity. For any $K>0$,

[TABLE]

First consider $(II)$. By computing the covariance structure, $(II)$ can be further reduced as follows: there exists some constant $C_{d}>0$, such that for $K$ large enough,

[TABLE]

Here in the last step we used standard estimates for Gaussian processes (via e.g. Dudley’s entropy integral in Lemma 7) and Gaussian concentration (cf. Lemma 8).

Next we consider $(I)$. Since the white noise processes involved in $(I)$ are independent for different $j$’s,

[TABLE]

Here $Z\equiv\min_{0\leq u\leq 1}\max_{0\leq v\leq 1}\mathbb{W}_{\{\kappa_{\ast},\ldots,d-1\}}([-u,v])$. Hence, for any $\mathcal{I}=\{S_{0},S_{1},\ldots,S_{J}\}$, the probability in question is bounded by

[TABLE]

Fix $\delta>0,\varepsilon>0$. Let $S_{j}\equiv 2^{2^{j}}\delta$ for $j\geq 1$. Then $J=\log_{2}\log_{2}(1/\delta)$. We choose $\delta>0$ such that $J$ is an even integer. For $j\geq 2$, we have

[TABLE]

The above inequality apparently holds for $j=1$ as $S_{0}=0$ by definition. Let $K\equiv\sqrt{C_{d}\log(C_{d}J/\varepsilon)}$. Then for $J$ large, the probability in question can be further bounded by

[TABLE]

If there exist $\varepsilon_{0},\tau_{0}>0$ such that $\mathbb{P}(Z\leq\tau_{0})\leq 1-\varepsilon_{0}<1$, the above probability vanishes as $J\to\infty$ (equivalently $\delta\to 0$) followed by $\varepsilon\to 0$, proving the lemma. Now we prove this claim. We focus on the case $\mathbb{W}_{2}$ to illustrate the main proof idea; the other cases follow from minor modifications but with substantial complications in notation. Note that for any $\tau>0$,

[TABLE]

Here $N,Y_{1},Y_{2},Y_{3}$ are independent random variables, where $N$ is distributed as standard normal, and $Y_{i}$’s are i.i.d. copies of $\sup_{0\leq v\leq 1}\lvert\mathbb{W}_{2}([0,v])\rvert$. The above probability is strictly less than $1$ as $\tau\to 0$, proving the claim. The proof is complete. ∎

Acknowledgments

The authors would like to thank four referees for offering a large number of suggestions that significantly improve the presentation of the article.