Set structured global empirical risk minimizers are rate optimal in general dimensions

TL;DR

This paper shows that in models with certain set structures, global empirical risk minimizers achieve optimal convergence rates even when traditional entropy-based bounds suggest sub-optimality, resolving longstanding doubts.

Contribution

It demonstrates that set structures in models enable global ERMs to be rate-optimal in general dimensions despite diverging entropy integrals.

Findings

Global ERMs are rate-optimal in models with set structures.

Matching upper and lower bounds are established for empirical process over measurable sets.

The results apply to image estimation, classification, isotonic regression, and density estimation.

Abstract

Entropy integrals are widely used as a powerful empirical process tool to obtain upper bounds for the rates of convergence of global empirical risk minimizers (ERMs), in standard settings such as density estimation and regression. The upper bound for the convergence rates thus obtained typically matches the minimax lower bound when the entropy integral converges, but admits a strict gap compared to the lower bound when it diverges. Birg\'e and Massart [BM93] provided a striking example showing that such a gap is real with the entropy structure alone: for a variant of the natural H\"older class with low regularity, the global ERM actually converges at the rate predicted by the entropy integral that substantially deviates from the lower bound. The counter-example has spawned a long-standing negative position on the use of global ERMs in the regime where the entropy integral diverges, as they are heuristically believed to converge at a sub-optimal rate in a variety of models. The present paper demonstrates that this gap can be closed if the models admit certain degree of `set structures' in addition to the entropy structure. In other words, the global ERMs in such set structured models will indeed be rate-optimal, matching the lower bound even when the entropy integral diverges. The models with set structures we investigate include (i) image and edge estimation, (ii) binary classification, (iii) multiple isotonic regression, (iv) $s$-concave density estimation, all in general dimensions when the entropy integral diverges. Here set structures are interpreted broadly in the sense that the complexity of the underlying models can be essentially captured by the size of the empirical process over certain class of measurable sets, for which matching upper and lower bounds are obtained to facilitate the derivation of sharp convergence rates for the associated global ERMs.

Equations

$$Y_i=f_0(X_i)+{\xi }_i,i=1,\dots ,n,$$

$${\hat{f}}_n\in \underset{f\in \mathcal{F}}{\mathrm{argmin}}\sum _{i=1}^n(Y_i-f(X_i){)}^2.$$

$${\int }_{c{\bar{r}}_n^2}^{{\bar{r}}_n}\sqrt{\log {\mathcal{N}}_{[]}(\epsilon ,\mathcal{F},L_2(P))}\mathrm{d}\epsilon \asymp \sqrt{n}\cdot {\bar{r}}_n^2.$$

$${\underline{r}}_n\sqrt{\log \mathcal{N}({\underline{r}}_n,\mathcal{F},L_2(P))}\asymp \sqrt{n}\cdot {\underline{r}}_n^2,$$

$$\log \mathcal{N}(\epsilon ,\mathcal{F},L_2(P))\asymp \log {\mathcal{N}}_{[]}(\epsilon ,\mathcal{F},L_2(P))\asymp {\epsilon }^{-2\alpha }.$$

$$\displaystyle\underline{r}_{n}\asymp n^{-\frac{1}{2(1+\alpha)}},\quad\bar{r}_{n}\asymp\big{(}n^{-\frac{1}{2(1+\alpha)}}\vee n^{-\frac{1}{4\alpha}}\big{)}\sqrt{\log^{\bm{1}(\alpha=1)}n}.$$

$$\mathbb{E}\underset{f\in \mathcal{F}-f_0:\Vert f{\Vert }_{L_2(P)}\le r_n}{\sup }|{\mathbb{G}}_n(f)|\asymp \sqrt{n}\cdot r_n^2.$$

$$n^{(\alpha -1)/2(\alpha +1)}\lesssim \mathbb{E}\underset{f\in \mathcal{F}}{\sup }|{\mathbb{G}}_n(f)|$$

$$\log {\mathcal{N}}_{[]}({\epsilon }^2,\mathcal{F},L_1(P))\lesssim {\epsilon }^{-2\alpha }.$$

$$\mathbb{E}\underset{C\in \mathcal{C}(\sigma )}{\sup }|{\mathbb{G}}_n(C)|\asymp \max \{{\sigma }^{1-\alpha },n^{(\alpha -1)/2(\alpha +1)}\}.$$

$$\mathbb{E}\underset{f\in \mathcal{F}}{\sup }|{\mathbb{G}}_n(f)|$$

$$\displaystyle\lesssim_{\alpha,p}\inf_{0\leq\gamma\leq\sigma/2}\bigg{\{}(\sigma\bm{1}_{\alpha<p\wedge 2}+\gamma\bm{1}_{\alpha>p\wedge 2})^{\{(p\wedge 2)-\alpha\}/2}+n^{-1/2}\sigma^{-\alpha}+\sqrt{n}\gamma\bigg{\}}$$

$$\asymp \{\begin{array}{ll}{\sigma }^{\{(p\wedge 2)-\alpha \}/2}+n^{-1/2}{\sigma }^{-\alpha },& \alpha <p\wedge 2;\\ n^{\frac{\alpha -p\wedge 2}{2(\alpha +2-p\wedge 2)}}+{\sigma }^{-\frac{\alpha -p\wedge 2}{2}}+n^{-1/2}{\sigma }^{-\alpha },& \alpha >p\wedge 2.\end{array}$$

$$\log {\mathcal{N}}_{[]}({\epsilon }^2,\mathcal{F},L_1(P)1_{\alpha >1}+L_2^2(P)1_{0<\alpha <1})\le L{\epsilon }^{-2\alpha }.$$

$$\displaystyle\mathbb{E}\sup_{f\in\mathcal{F}(\sigma)}\lvert\mathbb{G}_{n}(f)\rvert\lesssim_{\alpha}\max\big{\{}\sigma^{1-\alpha},n^{(\alpha-1)/2(\alpha+1)}\big{\}}.$$

$$\log \mathcal{N}(\epsilon /2,\mathcal{F}(\epsilon ),L_2(P))\ge L^{-1}{\epsilon }^{-2\alpha }.$$

$$\displaystyle\mathbb{E}\sup_{f\in\mathcal{F}(\sigma)}\lvert\mathbb{G}_{n}(f)\rvert\gtrsim_{\alpha}\max\big{\{}\sigma^{1-\alpha},n^{(\alpha-1)/2(\alpha+1)}\big{\}}.$$

$$\mathbb{E}\underset{f\in \mathcal{F}(\sigma )}{\sup }|{\mathbb{G}}_n(f)|$$

$${\epsilon }^{-2\alpha }\lesssim \log \mathcal{N}(\epsilon ,\mathcal{F}(2\epsilon ),L_2(P))\le \log {\mathcal{N}}_{[]}(\epsilon ,\mathcal{F},L_2(P))\lesssim {\epsilon }^{-2\alpha }.$$

$$\displaystyle\mathbb{E}\sup_{C\in\mathscr{C}(\sigma)}\lvert\mathbb{G}_{n}(C)\rvert\lesssim_{\alpha}\max\big{\{}\sigma^{1-\alpha},n^{(\alpha-1)/2(\alpha+1)}\}.$$

$$\displaystyle\mathbb{E}\sup_{C\in\mathscr{C}(\sigma)}\lvert\mathbb{G}_{n}(C)\rvert\gtrsim_{\alpha}\max\big{\{}\sigma^{1-\alpha},n^{(\alpha-1)/2(\alpha+1)}\}.$$

$$Y_i=1_{C_0}(X_i)+{\xi }_i,i=1,\dots ,n.$$

$${\hat{C}}_n\in \underset{C\in \mathcal{C}}{\mathrm{argmin}}\sum _{i=1}^n{\left(Y_i-1_C(X_i)\right)}^2.$$

$$\displaystyle P\lvert\widehat{C}_{n}\Delta C_{0}\rvert=P\big{(}\bm{1}_{\widehat{C}_{n}}-\bm{1}_{C_{0}}\big{)}^{2}=\int\big{(}\bm{1}_{\widehat{C}_{n}}-\bm{1}_{C_{0}}\big{)}^{2}\ \mathrm{d}P.$$

$$\displaystyle\mathbb{E}\sup_{C\in\mathscr{C}:P\lvert C\Delta C_{0}\rvert\leq\delta^{2}}\bigg{\lvert}\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\varepsilon_{i}(\bm{1}_{C}-\bm{1}_{C_{0}})(X_{i})\bigg{\rvert},$$

$$\underset{C_0\in \mathcal{C}}{\sup }{\mathbb{E}}_{C_0}P|{\hat{C}}_n\Delta C_0|\lesssim n^{-1/(\alpha +1)}.$$

$$\underset{C_0\in {\mathcal{C}}_d}{\sup }{\mathbb{E}}_{C_0}P|{\hat{C}}_n\Delta C_0|\lesssim n^{-2/(d+1)}.$$

$$Y_i=f_{C_0}(X_i){\eta }_i$$

$$Y_i=2a{f}_{C_0}(X_i)+{\xi }_i$$

$${\hat{C}}_n\equiv \underset{C\in \mathcal{C}}{\mathrm{argmin}}\sum _{i=1}^n(Y_i-2a{f}_C(X_i){)}^2.$$

$$\underset{C_0\in \mathcal{C}}{\sup }{\mathbb{E}}_{C_0}P|{\hat{C}}_n\Delta C_0|\lesssim n^{-1/(\alpha +1)}.$$

Full text

The present paper demonstrates that this gap can be closed if the models admit certain degree of ‘set structures’ in addition to the entropy structure. In other words, the global ERMs in such set structured models will indeed be rate-optimal, matching the lower bound even when the entropy integral diverges. The models with set structures we investigate include (i) image and edge estimation, (ii) binary classification, (iii) multiple isotonic regression, (iv) $s$-concave density estimation, all in general dimensions when the entropy integral diverges. Here set structures are interpreted broadly in the sense that the complexity of the underlying models can be essentially captured by the size of the empirical process over certain class of measurable sets, for which matching upper and lower bounds are obtained to facilitate the derivation of sharp convergence rates for the associated global ERMs.

The research of Q. Han is partially supported by NSF DMS-1916221.

1. Introduction

1.1. Overview

Empirical risk minimization (ERM) is one of the most widely used statistical procedures for the purpose of estimation and inference. Theoretical properties for various ERMs, in particular in terms of rates of convergence, have been intensively investigated by various authors [Bir83, BM93, BBM99, vdG87, vdG90, vdG93, vdG95, WS95, vdVW96, vdG00, Kol06], in a number of by-now standard settings. To motivate our discussion, let us focus on the standard Gaussian regression setting: Let $X_{1},\ldots,X_{n}$ be i.i.d. covariates taking value in $(\mathcal{X},\mathcal{A})$ with law $P$, and the responses $Y_{i}$’s are given by

[TABLE]

where $\xi_{i}$’s are i.i.d. $\mathcal{N}(0,1)$, and $f_{0}$ belongs to a uniformly bounded class $\mathcal{F}\subset L_{\infty}(1)$. One canonical global ERM in the regression model (1.1) is the least squares estimator (LSE):

[TABLE]

The performance of $\widehat{f}_{n}$ is usually evaluated through the risk under squared $L_{2}$ loss $\mathbb{E}_{f_{0}}\lVert\widehat{f}_{n}-f_{0}\rVert_{L_{2}(P)}^{2}$, or its ‘probability’ version.

The seminal work of Birgé and Massart [BM93] (and other references cited above) shows that an upper bound $\bar{r}_{n}^{2}$ for the risk $\mathbb{E}_{f_{0}}\lVert\widehat{f}_{n}-f_{0}\rVert_{L_{2}(P)}^{2}$ can be obtained by solving

[TABLE]

Here $\mathcal{N}_{[\,]}(\varepsilon,\mathcal{F},L_{2}(P))$ is the $\varepsilon$-bracketing number of $\mathcal{F}$ under $L_{2}(P)$. On the other hand, a lower bound $\underline{r}_{n}^{2}$ for the risk, often evaluated in a minimax framework, i.e. $\inf_{\widetilde{f}_{n}}\sup_{f_{0}\in\mathcal{F}}\mathbb{E}_{f_{0}}\lVert\widetilde{f}_{n}-f_{0}\rVert_{L_{2}(P)}^{2}\geq\underline{r}_{n}^{2}$, can be obtained (cf. [Bir83, YB99]) via a different equation

[TABLE]

where $\mathcal{N}(\varepsilon,\mathcal{F},L_{2}(P))$ is the $\varepsilon$-covering number of $\mathcal{F}$ under $L_{2}(P)$. Note that the left hand side of (1.2) is no smaller than the left hand side of (1.3), so we always have $\underline{r}_{n}\lesssim\bar{r}_{n}$. Suppose for now that the difference in the covering and bracketing entropy can be ignored, and it holds for some $\alpha>0$ that

[TABLE]

The parameter $\alpha>0$ measures the complexity of $\mathcal{F}$, and is closely related to the ‘smoothness’ of $\mathcal{F}$, cf. [vdV96a, vdG00, GN16]. Solving the equations (1.2) and (1.3) yields that

[TABLE]

Modulo the logarithmic factor in the boundary case $\alpha=1$, we see a somewhat strange phase-transition phenomenon:

• •

If $\alpha\in(0,1)$, $0<\operatorname*{\underline{lim}}_{n}\bar{r}_{n}/\underline{r}_{n}\leq\operatorname*{\overline{lim}}_{n}\bar{r}_{n}/\underline{r}_{n}<\infty$. In this regime, $\mathcal{F}$ is Donsker since a central limit theorem in $\ell^{\infty}(\mathcal{F})$ holds for the empirical process due to the convergence of the bracketing entropy integral at [math], cf. [vdVW96, Section 2.5.2].

• •

If $\alpha>1$, $\operatorname*{\underline{lim}}_{n}\bar{r}_{n}/\underline{r}_{n}=\infty$. In this regime, $\mathcal{F}$ is non-Donsker since there does not exist a central limit theorem in $\ell^{\infty}(\mathcal{F})$ for the empirical process—the limiting Brownian bridge process indexed by $\mathcal{F}$ is not sample bounded.

Although at this point (1.2) only gives an upper bound for $\bar{r}_{n}$, Birgé and Massart [BM93] showed by a stunning example that in the regime $\alpha>1$, $\bar{r}_{n}$ can actually be attained (up to logarithmic factors) for the global ERM (called ‘minimum contrast estimators’ therein) over a slightly constructed $\mathcal{F}$ based on Hölder classes on $[0,1]$ with smoothness less than $1/2$. Consequently, there is a genuine gap between the upper bound $\bar{r}_{n}$ and the lower bound $\underline{r}_{n}$ obtained from general empirical process techniques based on $L_{2}$ entropy structures (1.2) and (1.3) alone, in the regime $\alpha>1$.

The counter-example in [BM93] results in a long-standing negative position on the use of global ERMs in the regime $\alpha>1$, as they are heuristically believed to be rate-suboptimal in various problems falling into the non-Donsker regime, beyond the natural setting of Hölder-type smoothness classes, cf. [vdG00, SW10, Gun12, KS16], just to name a few references. A common (but perhaps vague) heuristic is that when $\alpha>1$, the class $\mathcal{F}$ is too ‘massive’ for global ERMs to achieve the optimal rate.

It should be mentioned that the rate sub-optimality phenomenon is due to the global nature of ERM that searches over the entire parameter space, since it is easy to construct a ‘theoretical’ rate-optimal estimator by searching over certain maximal packing sets of $\mathcal{F}$ even in the regime $\alpha>1$ (usually known as the ‘sieve’ estimator [Gre81, LC73]). For instance, the LSE over a maximal $\underline{r_{n}}$-packing set of $\mathcal{F}$ typically leads to the desired optimal rate of convergence. Such a theoretical construction often occurs in a minimax approach for a given statistical model, cf. [MT95, Gun12, Bru13].

At a deeper level from the perspective of empirical process theory, the upper bound (1.2) comes from the Dudley’s entropy integral, and the lower bound (1.3) is inherited with Sudakov minorization. From the recent work [Sau12, Cha14, vdGW17, HW19], it is now understood that the risk $r_{n}^{2}\equiv\mathbb{E}_{f_{0}}\lVert\widehat{f}_{n}-f_{0}\rVert_{L_{2}(P)}^{2}$ can be completely characterized (at least in the simple Gaussian regression model with uniformly bounded $\mathcal{F}$), by the following (not fully rigorous but essential)111Rigorously, $r_{n}$ is determined by the location of the maxima of the map $r\mapsto\sup_{f\in\mathcal{F}-f_{0}:\lVert f\rVert_{L_{2}(P)}\leq r}\mathbb{G}_{n}(\xi f-f^{2})-\sqrt{n}r^{2}$ provided it exists uniquely, cf. [vdGW17, HW19]. For Gaussian errors $\xi_{i}$’s and uniformly bounded $\mathcal{F}$, the order of $r_{n}$ can typically be obtained by matching upper and lower moment estimates for the LHS of (1.6). equation:

[TABLE]

Here $\mathbb{G}_{n}\equiv\sqrt{n}(\mathbb{P}_{n}-P)$ is the empirical process. Since Dudley’s entropy integral provides an upper bound, while the Sudakov minorization gives a lower bound, for the empirical process in (1.6) as soon as it enters the ‘Gaussian domain’ (= for $n$ large in our case), the only possibility for which $\bar{r}_{n}$ and $\underline{r}_{n}$ do not match lies in situations where the entropy integral bound deviates substantially from the Sudakov minorization. This is indeed the case in the non-Donsker regime $\alpha>1$: under a variant of the $L_{2}$ entropy condition (1.4) (see (2.7)), standard bounds lead to the estimates

[TABLE]

The upper and lower bounds above do not match, and neither of them can be improved without further structural assumptions. In particular, (1.7) leads to the discrepancy between $\bar{r}_{n}$ and $\underline{r}_{n}$ in (1.5) for non-Donsker $\mathcal{F}$’s.

Despite the strong suspicion in the literature (cited above) that the actual rate $r_{n}$ of global ERMs will likely match $\bar{r}_{n}$ which has a strict gap compared to the minimax lower bound $\underline{r}_{n}$, there appears recently some surprising special cases in which global ERMs are proved to be rate optimal even in the regime $\alpha>1$. One example is given by the multiple isotonic regression model studied by the author in [HWCS19]. When $d\geq 3$, by the entropy estimate in [GW07], the class of multiple isotonic functions is in the non-Donsker regime $\alpha>1$, but interestingly [HWCS19] proved that the natural LSE (= global ERM) is still minimax rate-optimal (up to logarithmic factors) in $L_{2}$ loss. The proof techniques in [HWCS19] are rather intricate and somewhat indirect, so they unfortunately do not shed light on why the LSE must be rate-optimal (see Remark 5.5 for more technical details).

The purpose of the present paper is to demonstrate a general underlying mechanism for the rate-optimality phenomenon for global ERMs beyond the isotonic LSE in general dimensions as mentioned above. This amounts to the identification of a sub-family of $\mathcal{F}$’s satisfying the $L_{2}$ entropy condition (1.4) (or see its variant (2.7)), in which the the associated global ERMs remain rate-optimal. As one may expect, the key step is to close the gap between the upper and lower bounds in (1.7) under suitable structural assumptions on $\mathcal{F}$, and for the purpose of rate-optimality, one should aim at improving the upper bound $n^{(\alpha-1)/2\alpha}$ to match the lower bound $n^{(\alpha-1)/2(\alpha+1)}$ in (1.7). We show (cf. Theorem 2.1) that this is indeed possible for $\mathcal{F}\subset L_{\infty}(1)$ in the non-Donsker regime $\alpha>1$, under a stronger $L_{1}$ entropy condition:

[TABLE]

An important class of $\mathcal{F}$ that verifies (1.8) is the class of indicators indexed by a class of measurable sets. More specifically, for a class of measurable sets $\mathscr{C}$, as the $L_{1}(P)$ size of any element $\bm{1}_{C}$ where $C\in\mathscr{C}$ is the same as its squared $L_{2}(P)$ size, (1.8) is automatically verified provided the $L_{2}$ entropy condition (1.4) (or (2.7)) is satisfied. Therefore, under the prescribed $L_{2}$ entropy condition alone, as long as the $L_{2}$-size of $\mathscr{C}$ is not too small, it holds for $\alpha\neq 1$ that

[TABLE]

Here $\mathscr{C}(\sigma)\equiv\{C\in\mathscr{C}:P(C)\leq\sigma^{2}\}$, and for a measurable set $C$, $\mathbb{G}_{n}(C)\equiv\mathbb{G}_{n}(\bm{1}_{C})$. For $\alpha>1$, the empirical process (1.9) is in the non-Donsker regime, and our estimate (1.9) is still sharp in this challenging regime thanks to the improved estimate of the generic bound (1.7) due to the $L_{1}$ entropy condition (1.8).

In light of (1.9), we will show that in models with certain “set structures”, the global ERMs will achieve the minimax optimal rates of convergence, i.e. $\bar{r}_{n}\asymp r_{n}\asymp\underline{r}_{n}$ (up to logarithmic factors) even in the regime $\alpha>1$ when the entropy integral diverges. Here “set structures” are interpreted broadly in the sense that the size of the underlying empirical process (1.6) indexed by $\mathcal{F}$ can be characterized by an empirical process indexed by certain class of measurable sets for which an estimate of the type (1.9) is possible. This concept will be illustrated throughout a detailed study on the risk behavior (or rates of convergence) for the natural global ERMs in the following models:

1. (1)

Image and edge estimation; 2. (2)

Binary classification; 3. (3)

Multiple isotonic regression (revisited); 4. (4)

$s$-concave density estimation,

all of which will be considered in general dimensions, where the problems necessarily fall into the non-Donsker regime $\alpha>1$. In the special case of the multiple isotonic regression model, our new techniques present a much easier and intuitive proof (compared to the previous work [HWCS19]) that explains the reason why the natural least squares estimator is indeed rate minimax (up to logarithmic factors) for $d\geq 3$—the complexity of the isotonic LSE is captured by that of the class of upper and lower sets that arise naturally in the min-max representation of the isotonic LSE, cf. [RWD88].

1.2. Related works

Prior upper bounds for the empirical process (1.9) in the regime $\alpha>1$ are obtained in, e.g. [Dud82, Theorem 2] or [Dud14, Theorem 11.4], with additional logarithmic factors and in the weaker ‘in probability’ form. This paper provides stronger matching upper and lower bounds for the expected supremum of the empirical process in (1.9), and in fact the bounds hold in much greater generality; see Theorem 2.1 for precise statements.

The major part of this work is based on Chapter 4 of the author’s University of Washington Ph.D. thesis in 2018. During the preparation the paper, the author becomes aware of the very nice work [DK19] which derives, among other things, global risk bounds for the log-concave (=[math]-concave) maximum likelihood estimators (MLEs) based on a reduction scheme of [CDSS18] and an upper bound similar to (1.9). Here we prove that the rate-optimality in example (4) holds for the maximum regime of $s$ in which the $s$-concave MLE exists. See Remark 3.9 for more technical remarks.

1.3. Organization

The rest of the paper is organized as follows. Section 2 is devoted to the new upper and lower bounds for the size of the empirical process indexed by a class of functions satisfying certain special entropy conditions that includes the class of measurable sets. Applications of these new bounds to the models mentioned above are detailed in Section 3. For clarity of presentation, proofs are deferred to Sections 4-5, and the Appendix.

1.4. Notation

For a real-valued random variable $\xi$ and $1\leq p<\infty$, let \lVert\xi\rVert_{p}:=\big{(}\mathbb{E}\lvert\xi\rvert^{p}\big{)}^{1/p} denote the ordinary $p$-norm.

For a real-valued measurable function $f$ defined on $(\mathcal{X},\mathcal{A},P)$, \lVert f\rVert_{L_{p}(P)}\equiv\lVert f\rVert_{P,p}\equiv\big{(}P\lvert f\rvert^{p})^{1/p} denotes the usual $L_{p}$-norm under $P$, and $\lVert f\rVert_{\infty}\equiv\sup_{x\in\mathcal{X}}\lvert f(x)\rvert$. $f$ is said to be $P$-centered if $Pf=0$. $L_{p}(g,B)$ denotes the $L_{p}(P)$-ball centered at $g$ with radius $B$. For simplicity we write $L_{p}(B)\equiv L_{p}(0,B)$.

Throughout the article $\varepsilon_{1},\ldots,\varepsilon_{n}$ will be i.i.d. Rademacher random variables independent of all other random variables. $C_{x}$ will denote a generic constant that depends only on $x$, whose numeric value may change from line to line unless otherwise specified. $a\lesssim_{x}b$ and $a\gtrsim_{x}b$ mean $a\leq C_{x}b$ and $a\geq C_{x}b$ respectively, and $a\asymp_{x}b$ means $a\lesssim_{x}b$ and $a\gtrsim_{x}b$ [$a\lesssim b$ means $a\leq Cb$ for some absolute constant $C$]. For two real numbers $a,b$, $a\vee b\equiv\max\{a,b\}$ and $a\wedge b\equiv\min\{a,b\}$. Slightly abusing notation, we write $\log(x)\equiv\log(e\vee x)$, and $\log\log(x)\equiv\log(e\vee\log(e\vee x))$.

2. Empirical processes indexed by sets

2.1. Setup and assumptions

Let $X_{1},\ldots,X_{n}$ be i.i.d. random variables with distribution $P$ on a sample space $(\mathcal{X},\mathcal{A})$, and $\mathscr{C}$ be a collection of measurable sets contained in $\mathcal{X}$. To avoid measurability digressions, we assume that $\mathscr{C}$ is countable throughout the article. For any $\sigma>0$, let $\mathscr{C}(\sigma)\equiv\{C\in\mathscr{C}:P(C)\leq\sigma^{2}\}$.

Following the standard notation for covering and bracketing numbers (cf. [vdVW96, pp. 83]), for a normed linear space $(\mathcal{F},\lVert\cdot\rVert)$, let the covering number $\mathcal{N}(\varepsilon,\mathcal{F},\lVert\cdot\rVert)$ be the minimum number of balls $\{g:\lVert g-f\rVert<\varepsilon\}$ of radius $\varepsilon$ under $\lVert\cdot\rVert$ needed to cover $\mathcal{F}$. Let the bracketing number $\mathcal{N}_{[\,]}(\varepsilon,\mathcal{F},\lVert\cdot\rVert)$ be the minimum number of $\varepsilon$-brackets under $\lVert\cdot\rVert$ needed to cover $\mathcal{F}$, where an $\varepsilon$-bracket $[\ell,u]\equiv\{f:\ell\leq f\leq u\}$ with $\lVert u-\ell\rVert<\varepsilon$. Following the notation in [Dud14, pp. 270, (7.4)], let $\mathcal{N}_{I}(\varepsilon,\mathscr{C},P)$ be the $\varepsilon$-bracketing number for $\mathscr{C}$ under $P$, i.e. $\mathcal{N}_{I}(\varepsilon,\mathscr{C},P)\equiv\mathcal{N}_{[\,]}(\varepsilon,\bm{1}_{\mathscr{C}},L_{1}(P))$ with $\bm{1}_{\mathscr{C}}\equiv\{\bm{1}_{C}:C\in\mathscr{C}\}$.

Assumption A*.*

Fix $\alpha>0$.

1. (E1)

$\log\mathcal{N}_{I}(\varepsilon,\mathscr{C},P)\leq L\varepsilon^{-\alpha}$. 2. (E2)

$\log\mathcal{N}(\varepsilon/4,\mathscr{C}(\sqrt{\varepsilon}),P)\geq L^{-1}\varepsilon^{-\alpha}.$

For examples satisfying the above entropy conditions, see e.g. [Dud14, Sections 8.3/8.4, or Theorem 8.3.2] on the class of upper/lower sets and convex bodies (cf. [Dud14, Theorem 8.4.1/Corollary 8.4.2]). $L$ will be a large enough absolute constant throughout the article, the dependence on which will not be explicitly stated in the theorems.

For $0<\alpha<1$, the bracketing condition in (E1) can also be replaced by a uniform entropy condition $\sup_{Q}\log\mathcal{N}(\varepsilon,\mathscr{C},Q)\leq L\varepsilon^{-\alpha}$, where the supremum is taken over all finitely discrete probability measures $Q$. Such a uniform entropy condition is satisfied if $\mathscr{C}$ is a VC-class (cf. [GN16, Section 3.6]).222[Bar16] advocates the notion of weak VC-major class as a generalization of VC-major class. The class of indicators over a class of sets $\mathscr{C}$ is weakly VC-major if and only if $\mathscr{C}$ is VC (cf. [Bar16, Definition 2.2]). This case is essentially covered in [GK06].

2.2. Upper and lower bounds

We first state the general upper and lower bounds for empirical processes indexed by general function classes satisfying certain entropy conditions.

Theorem 2.1**.**

1. (1)

Fix $p\geq 1$. Let $\mathcal{F}\subset L_{\infty}(1)$, and $\sup_{f\in\mathcal{F}}\lVert f\rVert_{L_{p}}\leq\sigma$. Suppose there exists some $\alpha>0$ such that for all $\varepsilon>0$, $\log\mathcal{N}_{[\,]}(\varepsilon,\mathcal{F},L_{p}(P))\leq L\varepsilon^{-\alpha}$. Then

[TABLE] 2. (2)

Suppose that the following entropy estimate holds for some $\alpha>0$:

[TABLE]

Then for $\sigma^{2}\gtrsim n^{-1/(\alpha+1)}$, $\alpha\neq 1$, we have:

[TABLE] 3. (3)

If in addition to (2.2), it holds that

[TABLE]

Then for $\sigma^{2}\gtrsim n^{-1/(\alpha+1)}$, $\alpha\neq 1$, we have:

[TABLE]

Here $\mathcal{F}(\sigma)\equiv\{f\in\mathcal{F}:Pf^{2}\leq\sigma^{2}\}$.

Proof.

See Section 4. ∎

Remark 2.2* (Comparison to classical bounds).*

By the standard local maximal inequality for the empirical processes (cf. [vdVW96, Lemma 2.14.3]), we have for $\sigma>0$ not too small,

[TABLE]

Suppose $\mathcal{F}\subset L_{\infty}(1)$ satisfies the $L_{2}$ entropy condition:

[TABLE]

In the Donsker regime $\alpha<1$, standard upper bound (2.6) and the lower bound (2.5) already match each other under (2.7). In the non-Donsker regime $\alpha>1$, these bounds lead to (1.7) whose upper and lower bounds do not match, both of which can be attained for certain instances of $\mathcal{F}$ satisfying (2.7). This means improvements for the generic bound (1.7) must require additional structural assumptions on $\mathcal{F}$. As the $L_{1}$ entropy condition (2.2) along with $\mathcal{F}\subset L_{\infty}(1)$ implies the rightmost part of (2.7), it follows that within the family of $\mathcal{F}\subset L_{\infty}(1)$’s satisfying the $L_{2}$ entropy condition (2.7) with $\alpha>1$, if in addition the stronger $L_{1}$ entropy condition (2.2) is satisfied, then the upper bound in (1.7) can be improved from $n^{(\alpha-1)/2\alpha}$ to $n^{(\alpha-1)/2(\alpha+1)}$ that matches the lower bound.

An important family of $\mathcal{F}$ satisfying the $L_{1}$ entropy condition (2.2) is the class of indicators over measurable sets. We formalize the results below.

Theorem 2.3**.**

Suppose (E1) holds and $\sigma^{2}\gtrsim n^{-1/(\alpha+1)}$, $\alpha\neq 1$. Then

[TABLE]

If furthermore (E2) holds, then

[TABLE]

Proof of Theorem 2.3.

(2.2) is verified using the fact that for any measurable set $C$, with $f\equiv\bm{1}_{C}$ we have $Pf=Pf^{2}$. (2.4) can be verified by noting that $\mathcal{N}_{I}(\varepsilon/4,\mathscr{C}(\sqrt{\varepsilon^{\prime}}),P)=\mathcal{N}_{[\,]}(\sqrt{\varepsilon}/2,\mathcal{F}(\sqrt{\varepsilon^{\prime}}),L_{2}(P))$ holds with $\mathcal{F}\equiv\{\bm{1}_{C}:C\in\mathscr{C}\}$ and any $\varepsilon,\varepsilon^{\prime}>0$ (and similarly for the covering number). ∎

Remark 2.4*.*

Some remarks on the upper bounds in Theorems 2.1 and 2.3:

1. (1)

Roughly speaking, the improved estimates (1) compared to the classical bound (2.6) come from careful $L_{p}$ chaining with bracketing that provides tighter controls at the finest resolution of the chaining step; see Section 4.1 for some heuristics and details. 2. (2)

It is also possible to consider the boundary case $\alpha=1$ in Theorem 2.3. Then the upper bound deviates from the lower bound by a logarithmic factor. In particular, suppose (E1)-(E2) hold and $\sigma^{2}\gtrsim n^{-1/2}$. Then $1\lesssim\mathbb{E}\sup_{C\in\mathscr{C}(\sigma)}\lvert\mathbb{G}_{n}(C)\rvert\lesssim\log n$.

Remark 2.5*.*

Some remarks on the lower bounds in Theorems 2.1 and 2.3:

1. (1)

The proof for the lower bound (2.5) is based on Gaussian randomization followed by an application of the multiplier inequality derived in the author’s previous work [HW19] that removes the effect of Gaussianization. This only requires some sharp upper bounds for the unconditional processes, as opposed to the approach of [GK06] using Rademacher minorization, which requires sharp upper bounds for conditional processes. 2. (2)

Condition (2.4) is also assumed in [GK06] under the name ‘$\alpha$-fullness’ (cf. Definition 3.3 therein). This condition is best verified on a case-by-case basis. For instance the $\alpha$-Hölder class on $[0,1]$ is $\alpha$-full, cf. the proof of Lemma 6 in [HW19].

3. Rate-optimal global ERMs

In this section, we apply the new bounds derived in the previous section to several models including (i) image and edge estimation, (ii) binary classification, (iii) multiple isotonic regression, and (iv) $s$-concave density estimation, all in general dimensions. Global ERMs in these models are non-Donsker problems in general dimensions, but we will show that in each of these models, the underlying empirical process problem (1.6) can be essentially characterized by an empirical process indexed by certain class of measurable sets. The bounds in Theorem 2.3 can then be used to prove that these global ERMs converge at an optimal rate (up to logarithmic factors), rather than a strictly sub-optimal rate as predicted using the entropy integral (= (1.2)) in [BM93]. Interestingly, Theorem 2.3 is typically applied without the localization. This is viable as the size of the expected supremum of empirical process is already diverging in the non-Donsker regime, so localization is usually uninformative.

3.1. Image estimation

Let $X_{1},\ldots,X_{n}$ be i.i.d. samples with law $P$ on a sample space $(\mathcal{X},\mathcal{A})$. In this subsection we consider the regression model:

[TABLE]

This model has been considered by [KT92, KT93] and more recently by [Bru13] under the name ‘image estimation’, cf. [KT93, Section 3.1], where $C_{0}$ is considered as the ‘image’, and $\mathcal{X}\setminus C_{0}$ is considered as ‘background’. We assume for simplicity that the $\xi_{i}$’s are i.i.d. $\mathcal{N}(0,1)$ and are independent of $X_{i}$’s. Let $\mathscr{C}$ be a collection of measurable sets in $\mathcal{X}$, and we will fit the regression model by $\{\bm{1}_{C}:C\in\mathscr{C}\}$. Our interest will be the behavior of the least squares estimator $\widehat{C}_{n}$ defined by

[TABLE]

We assume that $\widehat{C}_{n}$ is well-defined without loss of generality. We will measure the quality of $\widehat{C}_{n}$ via the expected symmetric difference of $\widehat{C}_{n}$ and $C_{0}$ under $P$ defined by

[TABLE]

By a relatively standard reduction (cf. Lemma 5.1), the risk of $\widehat{C}_{n}$ in symmetric difference can be related to the expected supremum

[TABLE]

and therefore we may apply Theorem 2.3 (or the more general Theorem 2.1). We formalize the result below.

Theorem 3.1**.**

Suppose that for some $\alpha\neq 1$, $\log\mathcal{N}_{I}(\varepsilon,\mathscr{C},P)\leq L\varepsilon^{-\alpha}$. Then

[TABLE]

Proof.

See Section 5. ∎

By [YB99], the rate $n^{-1/(\alpha+1)}$ cannot be improved in a minimax sense if furthermore a lower bound on the metric entropy on the same order as that of the upper bound is available.

As a straightforward corollary of the above Theorem 3.1, let $\mathscr{C}_{d}$ be the collection of all convex bodies contained in the unit ball in $\mathbb{R}^{d}$ and $P$ the uniform distribution on the unit ball.

Corollary 3.2**.**

Fix $d\geq 4$. Then

[TABLE]

Proof.

The claim essentially follows from [Dud14, Theorem 8.25, Corollary 8.26 ], asserting that we can take $\alpha=(d-1)/2$ in Theorem 3.1. ∎

The corollary shows that we can use a global least squares estimator rather than a sieved least squares estimator (cf. [Bru13]) to achieve the optimal rate of convergence.

Remark 3.3*.*

It is possible to impose certain tail conditions on the density of $P$ to extend the above corollary to a maximum risk bound over all convex sets in $\mathbb{R}^{d}$. In particular, the above result holds for any $P$ with compact support in $\mathbb{R}^{d}$ with a bounded Lebesgue density. A proof in this vein is carried out in the context of $s$-concave density estimation in $\mathbb{R}^{d}$ to be detailed ahead.

3.2. Edge estimation

In this subsection we consider the regression model studied in [KT93, MT95]:

[TABLE]

where $f_{C_{0}}(x)=2\bm{1}_{C_{0}}(x)-1$ and $\eta_{i}$’s are i.i.d. random variables such that $\mathbb{P}(\eta_{i}=1)=1/2+a$ and $\mathbb{P}(\eta_{i}=-1)=1/2-a$ for some known constant $a\in(0,1/2)$. Such a model is motivated by estimation of sets in multi-dimensional ‘black and white’ pictures, where $Y_{i}=1$ is interpreted as observing black, and $Y_{i}=-1$ is white. We refer the reader to [MT95] for more motivation for this model. The model (3.4) can be rewritten as

[TABLE]

where $\xi_{i}=f_{C_{0}}(X_{i})(\eta_{i}-2a)$’s are bounded errors. An important property for these errors is that $\mathbb{E}[\xi_{i}|X_{i}]=0$ for all $i=1,\ldots,n$. Note here $\xi_{i}$ is not independent of $X_{i}$ and hence a different analysis is needed. Now consider the least squares estimator

[TABLE]

A careful analysis to be detailed in Lemma 5.2 ahead shows that the risk of $\widehat{C}_{n}$ in symmetric difference (3.3) can still be related to the expected supremum of empirical process, so Theorem 2.3 is applicable in this setting as well. Formally, we have

Theorem 3.4**.**

Suppose that for some $\alpha\neq 1$, $\log\mathcal{N}_{I}(\varepsilon,\mathscr{C},P)\leq L\varepsilon^{-\alpha}$. Then

[TABLE]

Proof.

See Section 5. ∎

Compared to [MT95, Theorem 4.1], we use an unsieved least squares estimator to achieve the optimal rate, rather than their theoretical ‘sieved’ estimator. This provides another example for which the simple least squares estimator can be rate-optimal for non-Donsker function classes in a natural setting.

3.3. Binary classification: excess risk bounds

In this subsection we consider the binary classification problem in the learning theory, cf. [Tsy04, MN06]. Suppose one observes i.i.d. $(X_{1},Y_{1}),\ldots,(X_{n},Y_{n})$ with law $P$, where $X_{i}$’s take values in $\mathcal{X}$, and the responses $Y_{i}\in\{0,1\}$. A classifier $g:\mathcal{X}\to\{0,1\}$ over a class $\mathcal{G}$ has a generalization error $P(Y\neq g(X))$. The excess risk for a classifier $g$ over $\mathcal{G}$ under law $P$ is given by

[TABLE]

It is known that for a given law $P$ on $(X,Y)$, the minimal generalized error is attained by a Bayes classifier $g_{0}(x)\equiv\bm{1}_{\eta(x)\geq 1/2}$ where $\eta(x)\equiv\mathbb{E}[Y|X=x]$, cf. [DGL96]. It is then natural to consider an estimator of $g_{0}$ by minimizing the empirical training error:

[TABLE]

We assume $g_{0}\in\mathcal{G}$ for simplicity. The global ERM $\widehat{g}_{n}$ is previously studied in, e.g., [Tsy04, pp. 136], [MN06, pp. 2327], [Kol06, pp. 2627-2629], [GK06, pp. 1211-1213]. The quality of the estimator $\widehat{g}_{n}$ is measured by the excess risk:

[TABLE]

Let $\Pi$ be the marginal distribution of $X$ under $P$. We assume the following ‘Tsybakov’s margin(low noise) condition’ (cf. [MT99, Tsy04]): there exists some $c>0$ such that for all $g\in\mathcal{G}$,

[TABLE]

Here we have assumed that the margin condition holds with $\kappa=1$. Although faster rates are possible under more general margin condition $\kappa\geq 1$ (cf. [MT99, Tsy04]), we do not go into this direction to avoid distraction from our main points.

Below is the main result in this subsection, the formulation of which follows that of [Kol06, GK06].

Theorem 3.5**.**

Suppose $\mathcal{G}\equiv\{\bm{1}_{C}:C\in\mathscr{C}\}$ satisfies the following entropy condition: there exists some $\alpha\neq 1$ such that for all $\varepsilon>0$, $\log\mathcal{N}_{I}(\varepsilon,\mathscr{C},P)\leq L\varepsilon^{-\alpha}$. If $r_{n}^{2}\geq Kn^{-1/(\alpha+1)}$ for a large enough constant $K>0$, then

[TABLE]

holds for some constant $K^{\prime}>0$.

Proof.

See Section 5. ∎

Roughly speaking, the key to prove the above theorem is a control for the random variable

[TABLE]

where $\ell$ is the smallest integer such that $r_{n}^{2}2^{\ell}\geq 1$, and $\mathcal{F}_{j}\equiv\{\bm{1}_{y\neq g_{1}(x)}-\bm{1}_{y\neq g_{2}(x)}:\mathcal{E}_{P}(g_{1})\vee\mathcal{E}_{P}(g_{2})\leq r_{n}^{2}2^{j}\}$. A sharp estimate for the above random variable is achieved by an application of Theorem 2.3 and Talagrand’s inequality (cf. Appendix C).

Examples of $\mathcal{G}$ that satisfy the prescribed entropy conditions in the above theorem can be found in the comments after [MT99, Theorem 1, pp. 1813]. To put the above results in the literature, [Tsy04] considered the same problem under the working assumption $\alpha\in(0,1)$ (cf. [Tsy04, Assumption A2, pp. 140]). [MN06] used ratio-type empirical process techniques to give a more unified treatment of deriving risk bounds for this problem, when the class of classifiers satisfies a Donsker bracketing entropy condition (i.e. $0<\alpha<1$), or a Donsker uniform entropy condition. [GK06] further improved the result of [MN06] in the Donsker regime under a uniform entropy condition, by taking into account the size of the localized envelopes. See also [Kol06, pp. 2618], [Lec07, pp. 1706] for similar Donsker conditions. To the best knowledge of the author, our Theorem 3.5 gives a first result for the global ERM $\widehat{g}_{n}$ in (3.7) to be rate-optimal in the non-Donsker regime $\alpha>1$ in the classification problem.

3.4. Multiple isotonic regression

Let $X_{1},\ldots,X_{n}$ be i.i.d. with law $P$ on $[0,1]^{d}$. For simplicity we assume that $P$ is the uniform distribution on $[0,1]^{d}$. Consider the multiple isotonic regression model

[TABLE]

where $\xi_{i}$’s are i.i.d. Gaussian errors $\mathcal{N}(0,1)$, and $f_{0}\in\mathcal{M}_{d}\equiv\{f:[0,1]^{d}\to\mathbb{R},f(x)\leq f(y)\textrm{ for any }x\leq y\}$. Consider the isotonic least squares regression estimator $\widehat{f}_{n}$ defined via:

[TABLE]

The performance of $\widehat{f}_{n}$ in the multivariate setting is examined by [CGS18] for $d=2$ and [HWCS19] for $d\geq 3$. By the entropy estimate for uniformly bounded multiple isotonic functions in [GW07], $\mathcal{M}_{d}\cap L_{\infty}(1)$ is in the non-Donsker regime when $d\geq 3$, which is the main interesting case here.

Theorem 3.6**.**

Let $d\geq 2$. Then with $\gamma_{d,\textrm{iso}}=(2)\bm{1}_{d=2}+\bm{1}_{d\geq 3}$,

[TABLE]

Proof.

See Section 5. ∎

Compared to [HWCS19, Theorem 4], the above result gives improvements over logarithmic factors. The logarithmic factors in $d\geq 3$ are due to boundary behavior of $\widehat{f}_{n}$. For instance, if the errors are bounded, then we may remove these logarithmic factors to get a sharp rate $n^{-1/d}$ for $d\geq 3$. These logarithmic factors cannot be removed by the proof techniques in [HWCS19] even if the errors are bounded. The rate $n^{-1/d}$ is shown to be minimax optimal for squared $L_{2}$ loss in [HWCS19].

The proof of Theorem 3.6 contains two inter-related steps:

1. (1)

First, we show that with high enough probability,

[TABLE]

under the Gaussian noise assumption. 2. (2)

Second, using the first step, we show that

[TABLE]

where $\mathcal{L}_{d}$ is the collection of all upper and lower sets contained in $[0,1]^{d}$ (precise definition see the paragraph before the proof of Theorem 3.6 in Section 5). The expected supremum on the right hand side of the above display can be controlled using Theorem 2.3. Finally the claim follows by a standard reduction for the risk of LSE to expected supremum of empirical process.

Compared to the proof of [HWCS19, Theorem 4], the proof strategy described above is more informative by making a clear connection to the class $\mathcal{L}_{d}$ that drives the minimax rates of convergence for the multiple isotonic LSE. See also Remark 5.5 for a detailed technical comparison.

The approach outlined above can also be adapted to the problem of multivariate convex regression modulo technical difficulties due to unsolved boundary behavior of the convex LSE. See Remark 5.6 for some details.

3.5. $s$-concave density estimation in $\mathbb{R}^{d}$

We first introduce the class of $s$-concave densities on $\mathbb{R}^{d}$. The exposition follows that of [HW16]. Let

[TABLE]

A density $p$ on $\mathbb{R}^{d}$ is called $s$-concave, i.e. $p\in\mathcal{P}_{s}$ if and only if for all $x_{0},x_{1}\in\mathbb{R}^{d}$ and $\theta\in(0,1)$, p\big{(}(1-\theta)x_{0}+\theta x_{1}\big{)}\geq M_{s}(p(x_{0}),p(x_{1});\theta). It is easy to see that the densities $p$ have the form $p=\varphi_{+}^{1/s}$ for some concave function $\varphi$ if $s>0$, $p=\exp(\varphi)$ for some concave $\varphi$ if $s=0$, and $p=\varphi_{+}^{1/s}$ for some convex $\varphi$ if $s<0$. The function classes $\mathcal{P}_{s}$ are nested in $s$ in that for every $r>0>s$, we have $\mathcal{P}_{r}\subset\mathcal{P}_{0}\subset\mathcal{P}_{s}\subset\mathcal{P}_{-\infty}.$

Maximum likelihood estimation over $\mathcal{P}_{s}$ is proposed in [SW10], where existence and consistency of the MLE $\widehat{p}_{n}$ is proved. Global rates of convergence of the MLE $\widehat{p}_{n}$ over $\mathcal{P}_{s}$ is primarily studied in the special case $s=0$, also known as the log-concave MLE, cf. [KS16]. For general $s$-concave MLEs, the only result concerning global convergence rates is due to [DW16], who studied the univariate case $d=1,s>-1$, showing that $h^{2}(\widehat{p}_{n},p_{0})=\mathcal{O}_{\mathbf{P}}(n^{-4/5})$, where $h(\cdot,\cdot)$ is the Hellinger distance. Here we will be interested in general $s$-concave MLEs in general dimensions.

Theorem 3.7**.**

Suppose $s>-1/d$ and $d\geq 2$. Then

[TABLE]

where $\gamma_{d,s}=(2/3)\bm{1}_{d=2}+(2)\bm{1}_{d=3}+\bm{1}_{d\geq 4}$.

Proof.

See Section 5. ∎

The most interesting regime here is $d\geq 4$ when the entropy integral for the class of $s$-concave densities diverges. Modulo logarithmic factors, the rates of convergence for the $s$-concave MLE $\widehat{p}_{n}$ in squared Hellinger distance is $\mathcal{O}_{\mathbf{P}}(n^{-2/(d+1)})$, which matches the minimax lower bound for the smaller log-concave (= 0-concave) class, cf. [KS16]. During the preparation the paper, the author becomes aware of the very nice work [DK19] which derives, among other things, global risk bounds for the log-concave (i.e. $s=0$) MLEs in the Hellinger distance. The techniques used in both papers in this example share certain common features, while our general setting brings about further technical challenges. See Remark 3.9 below for more technical comments on the proof of Theorem 3.7.

The integrability restriction $s>-1/d$ is very natural in this setting: if $s<-1/d$, then there exists a family of $s$-concave densities with singularities so that the MLE does not exist. The following proposition makes this precise.

Proposition 3.8**.**

The $s$-concave MLE does not exist for $s<-1/d$.

Proof.

For $a\in\mathbb{R}^{d},b>0$, let $\widetilde{\varphi}_{a,b}(x)\equiv\lVert x-a\rVert\bm{1}_{\lVert x-a\rVert\leq b}+\infty\bm{1}_{\lVert x-a\rVert>b}$. Since $c_{b}\equiv\int\widetilde{\varphi}_{a,b}^{1/s}=\int\widetilde{\varphi}_{0,b}^{1/s}<\infty$ for $s<-1/d$, $p_{a,b}\equiv\widetilde{\varphi}_{a,b}^{1/s}/c_{b}$ is an $s$-concave density. The log likelihood function for observed $X_{1},\ldots,X_{n}$ is \ell(a,b)\equiv\log\prod_{i=1}^{n}p_{a,b}(X_{i})=\sum_{i=1}^{n}\big{[}(1/s)\log(\lVert X_{i}-a\rVert)-\log c_{b}\big{]} for $(a,b)$ such that $\max_{i}\lVert X_{i}-a\rVert\leq b$ and $X_{i}\neq a$ for $i=1,\ldots,n$. For $b$ large enough and $a$ approaches any of $X_{i}$’s, $\ell(a,b)\uparrow\infty$, so the MLE does not exist. ∎

The univariate case $d=1$ for the above proposition can also be found in [DW16].

Remark 3.9*.*

The proof of Theorem 3.7 relies on the following reduction scheme:

[TABLE]

where $\mathscr{C}_{d}$ is the class of convex bodies on $\mathbb{R}^{d}$.

The above reduction scheme (3.10) for $s=0$ is essentially achieved in [CDSS18], but a sharp bound for the expected supremum of the empirical process on the right hand side of the above display is not available therein. Here we show that the reduction (3.10) holds for the maximum regime in which the $s$-concave MLE exists. Once (3.10) is proven, the expected supremum on its right hand side can be controlled by Theorem 2.3 combined with a standard technique of ‘domain extension’ (cf. [vdV96b] or [vdVW96, Corollary 2.7.4]) under the envelope control (5.7). See the proof of Theorem 3.7 for more details.

4. Proofs for Section 2

We will prove Theorem 2.1 in this section. As (2) is a direct consequence of (1), we will prove (1) and (3) only.

4.1. Proof of Theorem 2.1-(1)

A heuristic way of seeing the bound (1) is the following. This requires some understanding for the proof of the classical maximal inequality (2.6):

• •

The entropy integral term $\int_{\gamma}^{\sigma}\sqrt{\log\mathcal{N}_{[\,]}(\varepsilon,\mathcal{F},L_{2}(P)}\,\mathrm{d}\varepsilon$ in (2.6) comes from $L_{2}$ chaining with bracketing in the ‘Gaussian regime’ using Bernstein’s inequality, starting from $\sigma$ down to $\gamma$.

• •

The residual term $\sqrt{n}\gamma$ in (2.6) comes from the bound $\lVert f\rVert_{L_{1}(P)}\leq\lVert f\rVert_{L_{2}(P)}$ towards the end level $\gamma$ of the $L_{2}$ chaining.

Now we wish to carry out $L_{p}$ chaining with bracketing for, say, $p\in[1,2]$. Clearly $\lVert f\rVert_{L_{2}(P)}\leq\lVert f\rVert_{L_{p}(P)}^{p/2}$ and $\lVert f\rVert_{L_{1}(P)}\leq\lVert f\rVert_{L_{p}(P)}$. This naturally hints the following conjecture: for $\sigma>0$ not ‘too small’,

[TABLE]

Below we implement this heuristic program rigorously and identify the regime of $\sigma>0$ in which the above bound holds.

Proof of Theorem 2.1-(1).

The proof is inspired by the proof of [vdVW96, Lemma 2.14.3] that is originated in [Oss87]. We use the same notation for convenience of the readers. Without loss of generality we assume $\sigma=2^{-q_{0}}$ for some $q_{0}\in\mathbb{N}$. By the assumption, there exist nested partitions $\{\mathcal{F}=\cup_{i=1}^{N_{q}}\mathcal{F}_{q,i}\}_{q=q_{0}}^{\infty}$ such that for all $q\geq q_{0}$, (i) $\max_{1\leq i\leq N_{q}}\lVert\sup_{f,g\in\mathcal{F}_{q,i}}\lvert f-g\rvert\rVert_{L_{p}}\leq 2^{-q}$, and (ii) $\log N_{q}\lesssim_{L,\alpha}2^{q\alpha}$. [Such nested partitions can be constructed as follows. First taking un-nested partitions $\{\mathcal{F}=\cup_{i=1}^{\bar{N}_{q}}\bar{\mathcal{F}}_{q,i}\}_{q=q_{0}}^{\infty}$ with $\bar{N}_{q}\leq\mathcal{N}_{[\,]}(2^{-q},\mathcal{F},L_{p})\leq\exp(L\cdot 2^{q\alpha})$. Then at level $q$, we use the partition consisting of all intersections of form $\{\cap_{r=q_{0}}^{q}\bar{\mathcal{F}}_{r,i_{r}}:1\leq i_{r}\leq\bar{N}_{r},q_{0}\leq r\leq q\}$. The first property above is obvious. The second one follows as $N_{q}\leq\prod_{r=q_{0}}^{q}\bar{N}_{r}$.]

Pick any $f_{q,i}\in\mathcal{F}_{q,i}$. For any $f\in\mathcal{F}$, let $\pi_{q}f\equiv f_{q,i}$ and $\Delta_{q}f\equiv\sup_{f,g\in\mathcal{F}_{q,i}}\lvert f-g\rvert$ if $f\in\mathcal{F}_{q,i}$. Let $a_{q}\equiv 2^{-q\{(p\wedge 2)+\alpha\}/2}$. Now define the indicator functions $A_{q-1}f\equiv\bm{1}_{\Delta_{q_{0}}f\leq\sqrt{n}a_{q_{0}},\ldots,\Delta_{q-1}f\leq\sqrt{n}a_{q-1}}$, $B_{q}f\equiv\bm{1}_{\Delta_{q_{0}}f\leq\sqrt{n}a_{q_{0}},\ldots,\Delta_{q-1}f\leq\sqrt{n}a_{q-1},\Delta_{q}f>\sqrt{n}a_{q}}$, and $B_{q_{0}}f\equiv\bm{1}_{\Delta_{q_{0}}f>\sqrt{n}a_{q_{0}}}$. In words, $A_{q-1}f$ indicates the region for which ‘Gaussian estimates’ are valid up to level $q-1$ for $f$, while $B_{q}$ indicates the region for which the ‘Gaussian estimate’ is first violated at level $q$ for $f$.

By the last display in [vdVW96, pp. 241], the following chaining holds for any $q_{1}>q_{0}$:

[TABLE]

We bound the expectation of the above four terms when applied with the empirical process, and name them $(I)$-$(IV)$. Roughly speaking, $(III)$ is the term with ‘Gaussian behavior’ due to the construction of $A_{q-1}f$, while for the terms $(I)$-$(II)$, the Gaussian estimate fails on $B_{q}f$ at level $q$. This will be compensated by the observation that the Gaussian estimate still holds at level $q-1$. The single term $(IV)$ terminates the chaining and will be handled via an $L_{p}$ control. Below we implement this rough idea precisely.

For $(I)$ and $(II)$, note that

[TABLE]

Hence

[TABLE]

For $q=q_{0}$, we use the trivial bound $\Delta_{q_{0}}fB_{q_{0}}f\leq 1$. For $q\geq q_{0}+1$, we will however implement control at level $q-1$: As the partitions are nested, $\Delta_{q}fB_{q}f\leq\Delta_{q-1}fB_{q}f\leq\sqrt{n}a_{q-1}$ for all $q\geq q_{0}+1$. In summary, for $q\geq q_{0}$,

[TABLE]

On the other hand,

[TABLE]

By Bernstein inequality, for any $q\geq q_{0}$,

[TABLE]

Furthermore,

[TABLE]

Combining (4.1)-(4.1), we have

[TABLE]

For $(III)$, we have Gaussian estimates as follows. Note that

[TABLE]

As the cardinality of $\{(\pi_{q}f-\pi_{q-1}f)A_{q-1}f:f\in\mathcal{F}\}$ is at most $N_{q}N_{q-1}\leq N_{q}^{2}$, by Bernstein inequality and a similar argument to (4.1),

[TABLE]

For $(IV)$, using similar arguments as in (4.1),

[TABLE]

Here in the last inequality we used $P\Delta_{q_{1}}fA_{q_{1}}f\leq\lVert\Delta_{q_{1}}fA_{q_{1}}f\rVert_{L_{p}}\leq 2^{-q_{1}}$. Finally, using Bernstein’s inequality again,

[TABLE]

Combining (4.1), (4.1)-(4.9), we obtain

[TABLE]

The $\lesssim$ does not depend on $q_{1}$. Now optimize over $q_{1}>q_{0}$ to conclude. ∎

Remark 4.1*.*

[Pol02] suggested the following key ‘recursive equality’ (cf. [Pol02, pp. 1043, last display])

[TABLE]

for the chaining method of [Oss87]. See [Pol02] for definitions of the above notation. Interestingly, (4.10) can also be viewed as a one-step chaining of (4.1) from $i$ to $i+1$, as we may regard $f-\pi_{i}f=R_{i}$, $A_{i}f=T_{i}$, $A_{i+1}f=T_{i}T_{i+1}$ and $B_{i+1}f=T_{i}T_{i+1}^{c}$ by translating the notation used here to those in [Pol02]. Some elementary algebra then reduces (4.1) to (4.10).

4.2. Proof of Theorem 2.1-(3)

The proof of (2.5) is divided into two steps:

1. (1)

First, we establish a lower bound for the Gaussianized empirical process; see Proposition 4.2. This can be done roughly via Sudakov minimization in the Gaussian regime, i.e., when $\sigma$ is not too small. 2. (2)

Second, we will control the Gaussianized empirical process by the standard symmetrized empirical process from above. This step requires several subtle estimates as a naive bound would incur additional undesirable logarithmic factors. This is done via the help of a multiplier inequality derived in [HW19] (cf. Appendix C). Roughly speaking, the logarithmic factors can be removed for the Gaussianized empirical process at sample size $n$ as long as the empirical process has size no smaller than Gaussian maxima along the ‘entire path’ from $1$ to $n$. Details see the proofs of Propositions 4.4 and 4.6.

We first prove the lower bound for Gaussianized empirical process.

Proposition 4.2**.**

Let $\mathcal{F}\subset L_{\infty}(1)$. For any $\sigma\geq 50n^{-1/2}$ such that

[TABLE]

we have

[TABLE]

Here $g_{1},\ldots,g_{n}$ are i.i.d. $\mathcal{N}(0,1)$.

Proof of Proposition 4.2.

By Sudakov minorization (cf. Lemma C.2), for any $\sigma>0$,

[TABLE]

We claim that for any $\sigma>0$ such that $\log\mathcal{N}(\sigma/4,\mathcal{F}(\sigma),L_{2}(P))\leq n\sigma^{2}/4000$,

[TABLE]

To see this, let $f_{1},\ldots,f_{N}$ be a maximal $\sigma/2$-packing set of $\mathcal{F}(\sigma)$ in the $L_{2}(P)$ metric, i.e. for $i\neq j$, $P(f_{i}-f_{j})^{2}\geq\sigma^{2}/4$. Since $P(f_{i}-f_{j})^{4}\leq 4P(f_{i}-f_{j})^{2}\leq 16\sigma^{2}$, we apply Bernstein’s inequality followed by a union bound to see that with probability at least $1-N^{2}\exp(-t)$,

[TABLE]

With $t=cn\sigma^{2}$ for a constant $c>0$ to be specified below, we obtain

[TABLE]

where $\mathcal{D}(\cdot,\cdot,\cdot)$ stands for the packing number. By choosing $c=1/10^{3}$ and $\log\mathcal{N}(\sigma/4,\mathcal{F}(\sigma),L_{2}(P))\leq n\sigma^{2}/4000$, we have

[TABLE]

This entails that $\mathcal{D}(\sigma/5,\mathcal{F}(\sigma),L_{2}(\mathbb{P}_{n}))\geq N\equiv\mathcal{D}(\sigma/2,\mathcal{F}(\sigma),L_{2}(P))$ with the above probability. Hence for any $\sigma>0$ such that $\log\mathcal{N}(\sigma/4,\mathcal{F}(\sigma),L_{2}(P))\leq n\sigma^{2}/4000$, with probability at least $1-e^{-n\sigma^{2}/2000}$,

[TABLE]

completing the proof of (4.12). Hence for any $\sigma\geq 50n^{-1/2}$ such that the entropy $\log\mathcal{N}(\sigma/4,\mathcal{F}(\sigma),L_{2}(P))\leq n\sigma^{2}/4000$, the claim of the proposition follows from (4.11) and (4.12). ∎

Next we eliminate the effect of the Gaussian multiplier. We need a technical lemma.

Lemma 4.3**.**

Let $g_{1},\ldots,g_{n}$ be i.i.d. $\mathcal{N}(0,1)$, and $\lvert g_{(n)}\rvert\leq\ldots\leq\lvert g_{(1)}\rvert$ be reversed order statistics of $\{\lvert g_{1}\rvert,\ldots,\lvert g_{n}\rvert\}$. Then there exists an absolute constant $K>0$ such that for any $c\in(0,K^{-1})$, and $0\leq t\leq K^{-1}\sqrt{\log(1/c)}$, we have \mathbb{P}\big{(}\lvert g_{(\left\lfloor cn\right\rfloor)}\rvert\leq t\big{)}\leq e^{-c^{2}n/K}.

Proof.

For notational convenience, we assume that $cn\in\mathbb{N}$. Let $\phi(t)\equiv\mathbb{P}(\lvert g_{1}\rvert>t)$. By [GN16, (2.23)], $\sqrt{2/\pi}\cdot\frac{t}{t^{2}+1}e^{-t^{2}/2}\leq\phi(t)\leq\min\{1,\sqrt{2/\pi}\cdot t^{-1}\}e^{-t^{2}/2}$. Let $t_{c}>0$ be such that $\phi(t_{c})=2c$. Then $t_{c}\leq\sqrt{2\log(1/2c)}$. By Bernstein’s inequality, for $0\leq t\leq t_{c}$, $2c\leq\phi(t)\leq 1$, so

[TABLE]

proving the claim. ∎

Proposition 4.4**.**

Let the conditions in Theorem 2.1-(3) hold for some $\alpha\in(0,1)$ and $L>0$ large enough. Then for $\sigma_{n}^{2}\geq cn^{-1/(\alpha+1)}$ with some constant $c>0$, $\mathbb{E}\sup_{f\in\mathcal{F}(\sigma_{n})}\lvert\mathbb{G}_{n}(f)\rvert\gtrsim_{\alpha}\sigma_{n}^{1-\alpha}$.

Proof.

By Proposition 4.2, the Gaussianized empirical process satisfies

[TABLE]

Suppose that $\sigma_{n}^{2}\leq c$. Without loss of generality, we assume that $\sigma_{n}^{2}\equiv\sigma_{n}(\gamma)^{2}=cn^{-\gamma}$ for some $0\leq\gamma\leq 1/(\alpha+1)$ and define $\sigma_{k}^{2}\equiv ck^{-\gamma}$. We first prove the following claim: there exists some $c_{1}\equiv c_{1}(c,\alpha)>0$ such that for any $0\leq\gamma\leq 1/(\alpha+1)$,

[TABLE]

To this end, let a_{n}\equiv(\sigma_{n}^{1-\alpha})^{-1}\mathbb{E}\sup_{f\in\mathcal{F}(\sigma_{n})}\big{\lvert}\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\varepsilon_{i}f(X_{i})\big{\rvert}. Then by the local maximal inequality (cf. Theorem 2.1-(1)), we see that $\sup_{k\in\mathbb{N}}a_{k}\leq C_{2}\equiv C_{2}(\alpha)$. Since $\sqrt{n}\sigma_{n}^{1-\alpha}=c^{(1-\alpha)/2}n^{\beta}$ where \beta\equiv\beta(\alpha,\gamma)\equiv\frac{1}{2}\big{(}1-(1-\alpha)\gamma\big{)}\in[\alpha/(1+\alpha),1/2], we have by Lemma C.4 that for a constant $c^{\prime}>0$ to be determined later,

[TABLE]

For $(I)$ in (4.2), using the same notation as in the proof of Lemma 4.3,

[TABLE]

by choosing c^{\prime}\equiv\exp\big{(}-C_{3}\log(4C_{1}C_{3})\big{)} and $n\geq C_{3}\log(4C_{1}C_{3})/(c^{\prime})^{2}$. On the other hand, for $(II)$ in (4.2), we have

[TABLE]

where G_{\alpha}=\int_{0}^{\infty}\big{(}\mathbb{P}(\lvert g_{1}\rvert>t)\big{)}^{\alpha/(1+\alpha)}\ \mathrm{d}t<\infty since Gaussian random variables have finite moments of any order, and the last inequality follows from Jensen’s inequality. Combining the above displays we see that

[TABLE]

where in the last inequality we used Jensen’s inequality. This proves our claim (4.13) by adjusting the constant and choosing $n\geq 2/c^{\prime}$. Now by de-symmetrization inequality (cf. [vdVW96, Lemma 2.3.6]), we have that

[TABLE]

For $\sigma_{n}\leq(c_{1}/4)^{1/\alpha}\wedge c^{1/2}$, the claim of the proposition follows from (4.13) and (4.15). On the other hand, the claim is trivial for $\sigma_{n}>(c_{1}/4)^{1/\alpha}\wedge c^{1/2}$. ∎

Remark 4.5*.*

From the proof of Proposition 4.4, the bracketing entropy upper bound is only used to prove $\sup_{k\in\mathbb{N}}a_{k}\leq C_{2}\equiv C_{2}(\alpha)$. This means that we may impose instead a uniform entropy upper bound condition as in [GK06] in the regime $0<\alpha<1$.

Proposition 4.6**.**

Let the conditions in Theorem 2.1-(3) hold for some $\alpha>1$ and $L>0$ large enough. Then for $\sigma_{n}^{2}\equiv cn^{-1/(\alpha+1)}$ with some constant $c>0$, $\mathbb{E}\sup_{f\in\mathcal{F}(\sigma_{n})}\lvert\mathbb{G}_{n}(f)\rvert\gtrsim n^{(\alpha-1)/2(\alpha+1)}.$

Proof.

Proposition 4.2 shows that

[TABLE]

Now applying Lemma C.4 in the following form,

[TABLE]

we see that for some $K>0$,

[TABLE]

On the other hand, by enlarging $K$ if necessary, Theorem 2.1-(1) entails that $\mathbb{E}\sup_{f\in\mathcal{F}(\sigma_{k})}\lvert\mathbb{G}_{k}(f)\rvert\leq K\cdot k^{(\alpha-1)/2(\alpha+1)}$, and hence

[TABLE]

by the assumption $\alpha>1$. Combining the upper and lower estimates we see that

[TABLE]

Now we will argue that the $\max$ operator can be ‘eliminated’. To this end, let $a_{k}\equiv\mathbb{E}\sup_{f\in\mathcal{F}(\sigma_{n})}\lvert\mathbb{G}_{k}(f)\rvert$ and $\beta\equiv(\alpha-1)/2(\alpha+1)$ for notational convenience. Let $k_{n}\equiv\operatorname*{arg\,max}_{1\leq k\leq n}a_{k}$. We claim that $k_{n}\in[cn,n]$ where $c=K^{-2/\beta}\in(0,1)$. To see this, we only need to note $K^{-1}n^{\beta}\leq\max_{1\leq k\leq n}a_{k}=a_{k_{n}}\leq Kk_{n}^{\beta}$, which entails $k_{n}^{\beta}\geq K^{-2}n^{\beta}$. Hence

[TABLE]

where the last inequality follows from Jensen’s inequality, proving the claim. ∎

Proof of Theorem 2.1-(3).

The claims follow by combining Propositions 4.4 and 4.6. ∎

5. Proofs for Section 3

5.1. Proof of Theorem 3.1

Before the proof of Theorem 3.1, we need following:

Lemma 5.1**.**

Consider the regression model (3.1) and the least squares estimator $\widehat{C}_{n}$ in (3.2). Suppose that $(\xi_{1},X_{1}),\ldots,(\xi_{n},X_{n})$ are i.i.d. random vectors with $\mathbb{E}[\xi_{1}|X_{1}]=0$ and $\mathbb{E}[\xi_{1}^{2}|X_{1}]\lesssim 1\vee\lVert\xi_{1}\rVert_{2}^{2}$ almost surely. Further assume that

[TABLE]

hold for some $\phi_{n}$ such that $\delta\mapsto\phi_{n}(\delta)/\delta$ is non-increasing. Then $\mathbb{E}_{C_{0}}P\lvert\widehat{C}_{n}\Delta C_{0}\rvert=\mathcal{O}(\delta_{n}^{2})$ holds for any $\delta_{n}\geq n^{-1/2}\max\{1,\lVert\xi_{1}\rVert_{2},\mathbb{E}^{1/8}\max_{1\leq i\leq n}\lvert\xi_{i}\rvert^{4}\}$ such that $\phi_{n}(\delta_{n})\leq\sqrt{n}\delta_{n}^{2}$, where the constant in $\mathcal{O}$ only depends on the constants in (5.1).

Proof.

See Appendix B. ∎

Proof of Theorem 3.1.

By Lemma 5.1 the risk of the least squares estimator

[TABLE]

can be solved by estimating the empirical processes in (5.1). Since the global entropy estimate is translation invariant (i.e. the metric entropy of $\{\bm{1}_{C}-\bm{1}_{C_{0}}:C\in\mathscr{C}\}$ is the same as that of $\{\bm{1}_{C}:C\in\mathscr{C}\}$), by Theorem 2.3, we obtain an estimate for the Rademacher randomized empirical process:

[TABLE]

It is now easy to see that the choice $\delta_{n}^{2}\asymp n^{-1/(\alpha+1)}$ leads to an upper bound of the above display on the desired order $\sqrt{n}\delta_{n}^{2}$. Note that the bound continues to hold when the left hand side of the above display is replaced with expected supremum without localization over $C\in\mathscr{C}$ for $P\lvert C\Delta C_{0}\rvert\leq\delta_{n}^{2}$. The Gaussian randomized empirical process can be handled via the multiplier inequality Lemma C.3 by letting $\psi_{n}(t)\equiv\psi(t)\equiv t^{\alpha/(\alpha+1)}$, whence

[TABLE]

completing the proof. ∎

5.2. Proof of Theorem 3.4

We need the following analogy of Lemma 5.1 before proving Theorem 3.4.

Lemma 5.2**.**

Consider the regression model (3.5) and the least squares estimator $\widehat{C}_{n}$ in (3.6). Further assume that

[TABLE]

holds for some $\phi_{n}$ such that $\delta\mapsto\phi_{n}(\delta)/\delta$ is non-increasing. Then $\mathbb{E}_{C_{0}}P\lvert\widehat{C}_{n}\Delta C_{0}\rvert=\mathcal{O}(\delta_{n}^{2})$ holds for any $\delta_{n}\geq n^{-1/2}\max\{1,\lVert\xi_{1}\rVert_{2},\mathbb{E}^{1/8}\max_{1\leq i\leq n}\lvert\xi_{i}\rvert^{4}\}$ such that $\phi_{n}(\delta_{n})\leq\sqrt{n}\delta_{n}^{2}$, where the constant in $\mathcal{O}$ only depends on the constants in the above inequality.

Proof.

See Appendix B. ∎

Proof of Theorem 3.4.

The proof follows by Lemma 5.2 and similar arguments as in the proof of Theorem 3.1. ∎

5.3. Proof of Theorem 3.5

We need some further notations to implement this program. For any $g\in\mathcal{G}$, write $f_{g}(x,y)\equiv\bm{1}_{y\neq g(x)}$. Let $\mathcal{G}(\delta)\equiv\{g\in\mathcal{G}:\mathcal{E}_{P}(g)\leq\delta\}$. Let $\ell$ be the smallest integer such that $r_{n}^{2}2^{\ell}\geq 1$, and for any $1\leq j\leq\ell$, let $\mathcal{F}_{j}\equiv\{f_{g_{1}}-f_{g_{2}}:g_{1},g_{2}\in\mathcal{G}(r_{n}^{2}2^{j})\}$.

Lemma 5.3**.**

Suppose $\mathcal{G}\equiv\{\bm{1}_{C}:C\in\mathscr{C}\}$ satisfies the same entropy condition as in Theorem 3.5. Then

[TABLE]

holds for some constants $K,K^{\prime}>0$ provided $r_{n}^{2}\cdot n^{1/(\alpha+1)}\geq K^{\prime\prime}$ for a large enough constant $K^{\prime\prime}>0$ depending on $c>0$ in (3.8) only.

Proof of Lemma 5.3.

By Talagrand’s concentration inequality (cf. Appendix C), with $\sigma_{j}^{2}\equiv\sup_{f\in\mathcal{F}_{j}}\lVert f\rVert_{L_{2}(P)}^{2}$,

[TABLE]

Let $\mathscr{S}\equiv\{S:f_{g}=\bm{1}_{S},g\in\mathcal{G}\}$. Note that for $g_{1}=\bm{1}_{C_{1}},g_{2}=\bm{1}_{C_{2}}\in\mathcal{G}$, where $C_{1},C_{2}\in\mathscr{C}$, we have $f_{g_{1}}=\bm{1}_{S_{1}},f_{g_{2}}=\bm{1}_{S_{2}}$, and hence

[TABLE]

This shows that $\mathcal{N}_{I}(\varepsilon,\mathscr{S},P)\leq\mathcal{N}_{I}(\varepsilon,\mathscr{C},P)$. Furthermore, for any $g\in\mathcal{G}(r_{n}^{2}2^{j})$, let $S\in\mathscr{S}$ be such that $f_{g}=\bm{1}_{S}$. Then similar to the above display, we have

[TABLE]

where the last inequality follows from the margin condition. Now by Theorem 2.3, we obtain

[TABLE]

On the other hand,

[TABLE]

This implies that with $s_{j}=s2^{j}$,

[TABLE]

Note that

[TABLE]

under the assumption. Now a union bound leads to the desired claim. ∎

Proof of Theorem 3.5.

Given the estimate in Lemma 5.3, the proof of the theorem closely follows that of [GK06, Theorem 7.1]. We provide some details for the convenience of the reader. On the event

[TABLE]

we have for any $g\in\mathcal{G}(r_{n}^{2}2^{j})\setminus\mathcal{G}(r_{n}^{2}2^{j-1})$ and $g^{\prime}\in\mathcal{G}(\sigma)$ for some $0<\sigma<r_{n}^{2}2^{j}$,

[TABLE]

Since $\sigma>0$ is taken arbitrarily, we see that on the event $E$, it holds that

[TABLE]

for all $g\in\mathcal{G}$ such that $\mathcal{E}_{P}(g)\geq r_{n}^{2}$. Furthermore, the above display entails that on the event $E$, we necessarily have $\mathcal{E}_{P}(\widehat{g}_{n})<r_{n}^{2}$ for $n$ large enough. Hence for any $g\in\mathcal{G}(r_{n}^{2}2^{j})\setminus\mathcal{G}(r_{n}^{2}2^{j-1})$, we have

[TABLE]

This entails that

[TABLE]

The proof of the claim is complete by combining (5.2)-(5.3) along with Lemma 5.3. ∎

5.4. Proof of Theorem 3.6

Lemma 5.4**.**

It holds for $C>0$ large enough that

[TABLE]

The $\mathcal{O}$ term is uniform in $f_{0}\in\mathcal{M}_{d}\cap L_{\infty}(1)$.

Proof.

Fix $f_{0}\in\mathcal{M}_{d}\cap L_{\infty}(1)$. By [HWCS19, Lemma 10, supplement], $\sup_{x\in[0,1]^{d}}(\widehat{f}_{n}-f_{0})(x)\leq\max_{1\leq i\leq n}Y_{i}+\lVert f_{0}\rVert_{\infty}\leq 2+\max_{1\leq i\leq n}\xi_{i}$. Hence with $Z_{n}\equiv\lVert\widehat{f}_{n}-f_{0}\rVert_{\infty}$, for $u$ large, $\mathbb{P}(Z_{n}>u\sqrt{\log n})\leq e^{-cu^{2}\log n}$ for some $c>0$. In particular, $\mathbb{E}Z_{n}^{4}\lesssim\log^{2}n$. Now the claim of the lemma follows by noting that

[TABLE]

for $C>0$ large. ∎

Recall that $B\subset\mathbb{R}^{d}$ is a lower (resp. upper) set if and only if for all $x\in B,y\in\mathbb{R}^{d}$ with $y_{i}\leq x_{i}$ (resp. $y_{i}\geq x_{i}$), $i=1,\ldots,d$, we have $y\in B$. Let $\mathcal{LL}_{d}$ be the collection of all upper and lower sets in $\mathbb{R}^{d}$ and $\mathcal{L}_{d}=\{B\cap[0,1]^{d}:B\in\mathcal{LL}_{d}\}$.

Proof of Theorem 3.6.

First consider $d\geq 3$. By Lemma 5.4, we only need to compute an upper bound for $\mathbb{E}_{f_{0}}\lVert\widehat{f}_{n}-f_{0}\rVert_{L_{2}(P)}^{2}\bm{1}_{\lVert\widehat{f}_{n}-f_{0}\rVert_{\infty}\leq C\sqrt{\log n}}\leq\bar{r}_{n}^{2}$. Similar to the proof of Lemma 5.1, this can be done by evaluating the size of two empirical processes

[TABLE]

Note that for any $f\in\mathcal{M}_{d}$,

[TABLE]

Here $\mathcal{L}_{d}$ is the class of all upper and lower sets in $[0,1]^{d}$. The last inequality follows since for any $f\in\mathcal{M}_{d}$, $\{f_{+}(x)>t\}=\{f(x)\vee 0>t\}\in\mathcal{L}_{d}$ and $\{f_{-}(x)>t\}=\{-(f(x)\wedge 0)>t\}=\{f(x)\wedge 0<-t\}\in\mathcal{L}_{d}$. Hence by [Dud14, Theorem 8.22], we may apply Theorem 2.3 with $\alpha=d-1$ to see that

[TABLE]

Using the multiplier inequality (cf. Lemma C.3) and contraction principle for empirical processes (cf. [HWCS19, Lemma 6, supplement]), we may further bound the two empirical processes in (5.4) by

[TABLE]

Solving (5.4) using the above inequality we obtain the rate $\bar{r}_{n}$ for $d\geq 3$.

For $d=2$, we may estimate the empirical process with an additional $\log n$:

[TABLE]

and the rate can be obtained similarly as above. ∎

Remark 5.5*.*

The proof for the analogue of Theorem 3.6 in [HWCS19], i.e. [HWCS19, Theorem 4], uses a completely different strategy. A rough argument is as follows. [HWCS19] first consider the problem $f_{0}=0$, where it is shown in Proposition 9 therein that for $\delta_{n}>0$ not too small,

[TABLE]

Then by a simple triangle inequality, if $d\geq 2$,

[TABLE]

Using the above inequality and (5.4), we obtain $\bar{r}_{n}^{2}\lesssim n^{-1/d}$ up to logarithmic factors. It is clear from the sketch here that the property of isotonic regression functions is only used in (5.5) where the problem is $f_{0}=0$. The proof for general $f_{0}\in L_{\infty}(1)$ in (5.6) is not very informative in the sense that the method of (5.6) is valid for any problem as long as one could solve the risk problem (= empirical process problem (5.5)) for one particular $f_{0}$. In contrast, the proof of Theorem 3.6 here shows that it is the complexity of the class of upper and lower sets $\mathcal{L}_{d}$ that leads to the minimax rate of convergence for the multiple isotonic LSE.

Remark 5.6*.*

It is possible to adapt the present approach to the problem of multivariate convex regression. The major difficulty here is to understand the boundary behavior for the convex LSE $\widehat{f}_{n}^{\mathrm{cvx}}$. In particular, if we can prove that the convex LSE $\widehat{f}_{n}^{\mathrm{cvx}}$ satisfies $\lVert\widehat{f}_{n}^{\mathrm{cvx}}-f_{0}\rVert_{\infty}=\mathcal{O}_{\mathbf{P}}(L_{n})$ for some slowly growing $L_{n}$ (in similar spirit to Lemma 5.4 for the isotonic LSE), then using similar arguments as in the proof of Theorem 3.6, we may conclude $\lVert\widehat{f}_{n}^{\mathrm{cvx}}-f_{0}\rVert_{L_{2}(P)}=\mathcal{O}_{\mathbf{P}}(n^{-1/(d+1)}\bar{L}_{n})$ for some slowly growing $\bar{L}_{n}$. Interestingly, recently [KGGS20] proved that under a fixed lattice design in a polytopal domain, the rate of the convex LSE can be improved to $n^{-2/(d+4)}$ for $d\leq 4$ and $n^{-1/d}$ for $d\geq 5$ (up to logarithmic factors) and these rates cannot be further improved in the worst case. These improved rates are due to reduced complexity of the class of convex functions on polytopal domains than those on smooth domains. Similar rates are obtained in [KGGS20] for bounded and Lipschitz convex LSEs under random designs. It remains open whether the unconstrained convex LSEs attain these rates over polytopal and more general domains under random designs, where the boundary behavior of $\widehat{f}_{n}^{\mathrm{cvx}}$ may play a crucial role.

5.5. Proof of Theorem 3.7

Proof of Theorem 3.7.

We only provide the proof for the most difficult case $-1/d<s<0$; the other cases are similar or simpler.

(Case 1: $d\geq 4$). We will relate the squared Hellinger distance $h^{2}(p_{0},\widehat{p}_{n})$ to that of the expected supremum of empirical process over the class of convex sets. The proof for this reduction is largely inspired by the idea of [CDSS18].

Using same arguments as in Step 1 of the proof of [DW16, Theorem 4.3], we may assume without loss of generality that $p_{0}\in\mathcal{P}_{s,M/2}$ and $\widehat{p}_{n}$ belongs to

[TABLE]

for some large $M$ with high probability. By the proof of [HW16, Lemma F.7] (especially (F.3) therein),

[TABLE]

Furthermore, it is not hard to see that $\widehat{p}_{n}$ is supported in the convex hull of $X_{1},\ldots,X_{n}$. By (5.7), $\kappa_{q}\equiv\mathbb{E}_{X\sim p_{0}}(1/p_{0}(X))^{q}=\int p_{0}^{1-q}<\infty$ for $q\in(0,1+sd)$. This means that $\mathbb{E}_{X\sim p_{0}}\max_{1\leq i\leq n}(1/p_{0}(X_{i}))^{q}\leq n\cdot\kappa_{q}$, so $\log\max_{i}(1/p_{0}(X_{i}))\leq C_{1}\log n$ with high probability for large $C_{1}>0$. Hence with $c_{n}\equiv n^{-C_{1}}$, $X_{1},\ldots,X_{n}\in\{p_{0}\geq c_{n}\}$ with high probability. Let $\widetilde{p}_{n}\equiv(\widehat{p}_{n}\vee c_{n})\bm{1}_{p_{0}\geq c_{n}}/\int(\widehat{p}_{n}\vee c_{n})\bm{1}_{p_{0}\geq c_{n}}$. Then with $b_{n}\equiv\int(\widehat{p}_{n}\vee c_{n})\bm{1}_{p_{0}\geq c_{n}}$, it follows that with high probability

[TABLE]

The last inequality in the second line of the above display follows as $\{x:\lVert x\rVert>(c_{n}/C_{M})^{s}\}\subset\{x:p_{0}(x)<c_{n}\}$ by (5.7), and therefore $\lvert\{p_{0}\geq c_{n}\}\rvert\subset\lvert\{x:\lVert x\rVert\leq(c_{n}/C_{M})^{s}\}\rvert\asymp_{d,M}c_{n}^{sd}$. As $s>-1/d$, by choosing $C_{1}>0$ large, we have $0\leq b_{n}-1\leq\mathcal{O}(n^{-1})$. This implies with high probability,

[TABLE]

On the other hand, let $\widetilde{p}_{0}\equiv p_{0}\bm{1}_{p_{0}\geq c_{n}}/\int p_{0}\bm{1}_{p_{0}\geq c_{n}}$ and $b_{0}\equiv\int p_{0}\bm{1}_{p_{0}\geq c_{n}}$. As $s>-1/d$, with $q\in(0,1+sd)$,

[TABLE]

This means by choosing $C_{1}>0$ large, we have $0\leq 1-b_{0}\leq\mathcal{O}(n^{-1})$. Hence

[TABLE]

and

[TABLE]

and

[TABLE]

Now by the integrability $\mathbb{E}_{P_{0}}\log^{2}p_{0}<\infty$, with $P_{0},\widetilde{P}_{0}$ denoting the distributions of $p_{0},\widetilde{p}_{0}$, it follows that with high probability,

[TABLE]

Here $\mathscr{C}_{d}$ is the set of all convex bodies in $\mathbb{R}^{d}$. The inequality ($\ast$) follows as for any $s$-concave density $p$, $\{(\log p(x))_{+}\geq t\}=\{\log p(x)\vee 0\geq t\}=\{p(x)\vee 1\geq e^{t}\}=\{\varphi(x)\wedge 1\leq e^{st}\}$ and \{(\log p(x))_{-}\leq t\}=\{-\big{(}\log p(x)\wedge 0\big{)}\leq t\}=\{p(x)\wedge 1\geq e^{-t}\}=\{\varphi(x)\vee 1\leq e^{-ts}\} are convex sets, and $-C_{1}\log n\leq\log c_{n}\leq\log\widetilde{p}_{n}\leq\log(M)$ for $n$ large. The inequality ($\ast\ast$) follows from (5.11).

Hence we only need to bound the entropy $\mathcal{N}_{I}(\varepsilon,\mathscr{C}_{d},P_{0})$. To this end, for a multi-index $\ell=(\ell_{1},\ldots,\ell_{d})\in\mathbb{Z}_{\geq 0}^{d}$, let $I_{\ell}\equiv\prod_{k=1}^{d}[2^{\ell_{k}}-1,2^{\ell_{k}+1}-1]$. Then $\lvert I_{\ell}\rvert\asymp 2^{\sum_{k}\ell_{k}}$. Let $\{(A_{\ell,j},B_{\ell,j}):1\leq j\leq N_{\ell}\}$ be an $\varepsilon_{\ell}$-bracket for $\{C|_{I_{\ell}}:C\in\mathscr{C}\}$ under the Lebesgue measure. By [Dud14, Theorem 8.25], we have $\log N_{\ell}\lesssim_{d}(\lvert I_{\ell}\rvert^{-1}\varepsilon_{\ell})^{(1-d)/2}$. Let $\varepsilon_{\ell}=a_{\ell}\cdot\varepsilon$, where $a_{\ell}\equiv\lvert I_{\ell}\rvert^{(1+\delta)}$ for some $\delta>0$ such that $1<1+\delta<(-sd)^{-1}\in(1,\infty)$. Then $\{(\sum_{\ell}\bm{1}_{A_{\ell,j_{\ell}}}\bm{1}_{I_{\ell}},\sum_{\ell}\bm{1}_{B_{\ell,j_{\ell}}}\bm{1}_{I_{\ell}}):1\leq j_{\ell}\leq N_{\ell},\ell\in\mathbb{Z}_{\geq 0}^{d}\}$ forms a bracket for $\mathscr{C}\cap\mathbb{R}_{\geq 0}^{d}$ with $P_{0}$-size

[TABLE]

The logarithm of the number of the brackets can be bounded by

[TABLE]

Other quadrants can be handled similarly. This means that $\log\mathcal{N}_{I}(\varepsilon,\mathscr{C},P_{0})\lesssim\varepsilon^{(1-d)/2}$, and hence Theorem 2.3 applies to (5.5).

(Case 2: $d=3$). The situation for $d=3$ is similar; but with an additional $\log n$ term in the estimate for the empirical process $\mathbb{E}\sup_{C\in\mathscr{C}_{3}}\lvert\mathbb{G}_{n}(C)\rvert\lesssim\log n$ (cf. Remark 2.4), and therefore the rate in squared Hellinger comes with an additional $\log n$.

(Case 3: $d=2$). We employ an idea in [DW16] in $d=1$, which first calculates the $L_{2}$ entropy of the class of bounded $s$-concave functions on $[0,1]$ by discretization of the range of the underlying convex functions until a prescribed $L_{2}$ error is reached at the level of $s$-concave densities, and then use the integrability of $\mathcal{P}_{s,M}$ to extend the brackets from $[0,1]$ to $\mathbb{R}$. Our arguments below substantially simplify those presented in [DW16]. A similar idea is exploited in [KS16] in $d=2,3$ in the context of log-concave densities, with further technicalities due to the unknown shapes of domain at the truncated levels in dimensions $2$ and $3$ (for $d=1$ they are simply intervals).

Let $\widetilde{\mathcal{P}}_{s}(I,B)\equiv\{p\textrm{ is s-concave on }I\subset\mathbb{R}^{2}:0\leq p(x)\leq B,\forall x\in I\}$. We write $\widetilde{\mathcal{P}}_{s}=\widetilde{\mathcal{P}}_{s}([0,1]^{2},1)$ for simplicity. We claim that for $s>-1$,

[TABLE]

Fix $\varepsilon>0$. Let $y_{k}\equiv 2^{k},1\leq k\leq k_{0}$, where $k_{0}$ is the smallest integer such that $y_{k_{0}}^{1/s}\leq\varepsilon$, i.e. $k_{0}=\lceil\log_{2}((1/\varepsilon)^{-s})\rceil$. Let $\{(A_{j},B_{j}):A_{j}\supset B_{j}\}_{j=1}^{N_{1}}$ be an $\varepsilon_{0}^{2}$-bracket for all convex sets in $[0,1]^{2}$ under the Lebesgue measure. By [Dud14, Theorem 8.25, Corollary 8.26], we have $\log N_{1}\lesssim\varepsilon_{0}^{-1}$. By [KS16, Proposition 4, supplement], for each $j=1,\ldots,N_{1}$, and $k=1,\ldots,k_{0}$, we may find a lower $\varepsilon_{j,k}$-bracket $\{\underline{f}_{j,k,m}:1\leq m\leq\underline{N}_{j,k}\}$ in $L_{2}$ (resp. upper $\varepsilon_{j,k}$-bracket $\{\bar{f}_{j,k,m}:1\leq m\leq\bar{N}_{j,k}\}$ in $L_{2}$) for non-negative convex functions defined on $B_{j}$ with an upper bound $2^{k}$, such that $\log(\bar{N}_{j,k}\vee\underline{N}_{j,k})\lesssim(2^{k}/\varepsilon_{j,k})\log(2^{k}/\varepsilon_{j,k})$.

For any $p\in\widetilde{\mathcal{P}}_{s}$, let $\varphi=p^{s}$ be the underlying convex function. Let $C_{k}\equiv\{\varphi\leq y_{k}\}$. Let $(A_{j_{k}},B_{j_{k}}),A_{j_{k}}\supset B_{j_{k}}$ be a bracket for $A_{j_{k}}\supset C_{k}\supset B_{j_{k}}$, and let $\underline{f}_{j_{k},k,m}$ (resp. $\bar{f}_{j_{k},k,m}$) be a lower (resp. upper) bracket for ${\varphi}|_{B_{j_{k}}}$.

Let $A_{j_{0}}\equiv\emptyset$ and $y_{0}\equiv 1$. Consider an upper bracket for $p$ of form

[TABLE]

and a lower bracket of $p$ of form \sum_{k=1}^{k_{0}}\big{[}\big{(}\bar{f}_{j_{k},k,m}\wedge y_{k}\big{)}\bm{1}_{B_{j_{k}}\setminus A_{j_{k-1}}}\big{]}^{1/s}. For the choice $\varepsilon_{0}\equiv\varepsilon$ and $\varepsilon_{j,k}\equiv\varepsilon\cdot 2^{2k}$, this bracket has squared $L_{2}$ size bounded, up to a constant depending only on $s$, by

[TABLE]

The logarithm of the total number of brackets can be bounded by

[TABLE]

proving the claim (5.13). Let $I_{\ell}$ be the same as in the proof for $d\geq 4$. By rescaling, it follows that

[TABLE]

By (5.7), on $I_{\ell}$, $\sup_{x\in I_{\ell}}\sup_{p\in\mathcal{P}_{s,M}}p(x)\leq\lvert I_{\ell}\rvert^{1/2s}$. Let $b_{\ell}=\lvert I_{\ell}\rvert^{-\delta^{\prime}}$ for some $\delta^{\prime}\in(0,(-1/s-1)/2)$, and $\{\underline{f}_{j,\ell},\bar{f}_{j,\ell}:1\leq j\leq N_{\ell}\}$ be a $b_{\ell}\varepsilon$-bracket for $\mathcal{P}_{s,M}|_{I_{\ell}}$ under $L_{2}$. A global bracket for $\mathcal{P}_{s,M}$ can be obtained by assembling these local brackets for all(=four) quadrants, with squared $L_{2}$-size at most $\varepsilon^{2}\sum_{\ell}b_{\ell}^{2}\lesssim\varepsilon^{2}$, and the logarithm of the number of brackets is

[TABLE]

Hence for $s>-1/2$, $\log\mathcal{N}_{[\,]}(\varepsilon,\mathcal{P}_{s,M},h)=\log\mathcal{N}_{[\,]}(\varepsilon,\mathcal{P}_{2s,M},L_{2})\lesssim\varepsilon^{-1}\log(1/\varepsilon)$. The rest of the proof is a standard computation of the size of the localized empirical process via Hellinger bracketing numbers (cf. [vdVW96, Theorem 3.4.4]), so we omit the details. ∎

Appendix A Upper and lower bounds for weighted empirical processes

As a direct application of Theorem 2.3, we consider upper and lower bounds for ratio-type empirical processes. Such bounds are initiated in [Wel78, SW82, Stu82, MSW83, Stu84] for uniform empirical processes on (subsets of) $\mathbb{R}$ (or $\mathbb{R}^{d}$), and are further investigated in [Ale87] for VC classes of sets, and extended by [GKW03, GK06] who studied more general VC-subgraph classes. These authors work with classes satisfying uniform entropy conditions, and the class of sets (or functions) need to be Donsker apriori. The lack of corresponding results for non-Donsker class of sets are mainly due to the lack of sharp upper and lower bounds for the behavior of the empirical process. Here we fill in this gap by using Theorem 2.3.

Theorem A.1**.**

Let $r_{n}^{2}\gtrsim n^{-1/(\alpha+1)}$ and \gamma_{n}\equiv n^{1/2}r_{n}\big{(}r_{n}^{\alpha-1}\wedge n^{-\frac{\alpha-1}{2(\alpha+1)}}\big{)}. Then we have the following:

1. (1)

If (E1) holds and $r_{n}^{2\alpha}\log\log n\to 0$,

[TABLE] 2. (2)

If (E1)-(E2) hold,

[TABLE]

Theorem A.2**.**

Let $r_{n}^{2}\gtrsim n^{-1/(\alpha+1)}$ and there exists some large constant $K_{\alpha}>0$ such that:

1. (1)

If (E1) holds and $\operatorname*{\underline{lim}}_{n\to\infty}r_{n}^{2}\cdot n^{1/(\alpha+1)}\geq\underline{\rho}$ for some $\underline{\rho}\in(K_{\alpha},\infty]$, then

[TABLE] 2. (2)

If furthermore (E2) holds and $\operatorname*{\overline{lim}}_{n\to\infty}r_{n}^{2}\cdot n^{1/(\alpha+1)}\leq\bar{\rho}$ for some $\bar{\rho}\in(K_{\alpha},\infty]$, then

[TABLE]

Remark A.3*.*

Some technical remarks:

1. (1)

An interesting corollary of Theorem A.2 is that under entropy conditions (E1)-(E2), the sequence in the theorem converges to [math] as $n\to\infty$ almost surely if and only if $r_{n}^{2}\cdot n^{1/(\alpha+1)}\to\infty$. 2. (2)

Theorems A.1 and A.2 are also valid in their $L_{p}(1\leq p<\infty)$ versions (which can be seen by integrating the tail estimates in the proofs). For instance, if (E1) holds, then

[TABLE]

and

[TABLE] 3. (3)

We may consider more general weighting functions of form $\phi(\sqrt{P(C)})$ as in [GK06] rather than the special cases $\phi_{1}(t)=t$ in Theorem A.1 and $\phi_{2}(t)=t^{2}$ in Theorem A.2. Here we make these choices mainly due to the fact that $\phi_{1},\phi_{2}$ are of special interest in the history of empirical process theory [Wel78, SW82, Stu82, MSW83, Stu84, Ale87], and the corresponding results for more general cases follow from minor modifications of the proofs. 4. (4)

It is also straightforward to consider corresponding ratio limit theorems for function classes satisfying the conditions of Theorem 2.1; we omit these digressions.

We will investigate the behavior of ratio-type empirical processes in a more general setting as in [GK06]. Let $\phi$ be a continuous and strictly increasing function with $\phi(0)=0$. Let $\mathscr{C}(r)\equiv\{C\in\mathscr{C}:P(C)\leq r^{2}\}$ and $\mathscr{C}(r,s]\equiv\mathscr{C}(s)\setminus\mathscr{C}(r)$. Fix $0<r<\delta\leq 1$. For a real number $1<q\leq 2$, let $\ell\equiv\ell_{r,\delta,q}$ be the smallest integer no smaller than $\log_{q}(\delta/r)$. For any $\bm{s}\equiv(s_{1},\ldots,s_{\ell})\in\mathbb{R}_{\geq 0}^{\ell}$, let

[TABLE]

The following result is essentially due to [GK06]. We state a somewhat simplified and easier-to-use version.

Proposition A.4**.**

Assume that $\phi$ is continuous, strictly increasing and satisfies $\sup_{r\leq x\leq 1}\phi(qx)/\phi(x)=\kappa_{r,q}<\infty$ for some $1<q\leq 2$. Then for any $\bm{s}\equiv(s_{1},\ldots,s_{\ell})\in\mathbb{R}_{\geq 0}^{\ell}$, both the probabilities

[TABLE]

and

[TABLE]

can be bounded by K\sum_{j=1}^{\ell}\exp\big{(}-s_{j}/K\big{)}. Here $K>0$ is a universal constant.

Proof of Proposition A.4.

We only prove the first claim; the second follows from similar arguments. The proof is a simple application of Talagrand’s concentration inequality combined with a peeling device. Write $\mathscr{C}_{j}\equiv\mathscr{C}(rq^{j-1},rq^{j}]$ and $\phi_{q}(u)\equiv\phi(rq^{j})$ if $u\in(rq^{j-1},rq^{j}]$ for notational convenience. By Talagrand’s concentration inequality,

[TABLE]

where $\sigma_{j}^{2}=\sup_{f\in\mathscr{C}_{j}}P(C)=r^{2}q^{2j}$. Hence by a union bound we see that with probability at least $1-\sum_{j=1}^{\ell}K\exp(-s_{j}/K)$, it holds that

[TABLE]

Now the conclusion follows from $\sup_{r\leq x\leq 1}\phi(qx)/\phi(x)<\infty$. ∎

The next lemma, due to [Ale87, Lemma 7.2], provides a convenient device to derive almost sure results for ratio-type empirical processes. For any measurable function $f$, let $\sigma_{P}f\equiv\sqrt{Pf^{2}}$.

Lemma A.5**.**

Let $c_{n},u_{n}$ be such that $c_{n}/n\downarrow$ and $u_{n}\downarrow$, and assume that $r_{n}\downarrow$ and $\sqrt{n}\delta_{n}\uparrow$. For a centered function class $\mathcal{F}\subset L_{2}(P)$, let

[TABLE]

and

[TABLE]

Assume that $\inf_{n\geq 1,t\in[r_{n},\delta_{n}]}c_{n}\frac{\phi(t)}{t}>0$. Then if $\mathbb{P}(A_{n}^{\varepsilon})=\mathcal{O}(1/(\log n)^{1+\theta})$ holds for some $\varepsilon,\theta>0$, we have $\mathbb{P}(A_{n}\textrm{ i.o.})=0$.

Proof of Theorem A.1 .

Consider the first claim. Note that for $0<\alpha<1$, $\beta_{n,q}\lesssim\max_{1\leq j\leq\ell}\frac{(r_{n}q^{j})^{1-\alpha}}{r_{n}q^{j}}\asymp r_{n}^{-\alpha}$, while for $\alpha>1$, $\beta_{n,q}\lesssim\max_{1\leq j\leq\ell}\frac{n^{(\alpha-1)/2(\alpha+1)}}{r_{n}q^{j}}\asymp n^{\frac{\alpha-1}{2(\alpha+1)}}/r_{n}$. For $s_{j}\equiv s+2K\log j$, we have

[TABLE]

and the probability estimate $K\sum_{j=1}^{\ell}\exp(-s_{j}/K)=Ke^{-s}\sum_{j=1}^{\ell}j^{-2}\leq K^{\prime}e^{-s}$. This proves that

[TABLE]

The first claim of (1) follows from Lemma A.5 by setting $s\asymp\log\log n$, and requiring $\beta_{n,q}\gg\sqrt{\log\log n}\vee\log\log n\cdot n^{-\alpha/(2(\alpha+1))}$. The second claim follows from similar lines by observing that under (E2), Theorem 2.3 yields that for $0<\alpha<1$, $\beta_{n,q}\gtrsim\max_{1\leq j\leq\ell}\frac{(r_{n}q^{j})^{1-\alpha}}{r_{n}q^{j}}\asymp r_{n}^{-\alpha}$, while for $\alpha>1$, $\beta_{n,q}\gtrsim\max_{1\leq j\leq\ell}\frac{n^{(\alpha-1)/2(\alpha+1)}}{r_{n}q^{j}}\asymp n^{\frac{\alpha-1}{2(\alpha+1)}}/r_{n}$, and $\tau_{n,q}$ can be estimated from above using the same arguments. ∎

Proof of Theorem A.2.

The proof of Theorem A.2 uses a similar strategy as that of Theorem A.1. For convenience of the reader we provide some details. Consider the first claim. Note that for $0<\alpha<1$, $\beta_{n,q}\lesssim\max_{1\leq j\leq\ell}\frac{(r_{n}q^{j})^{1-\alpha}}{r_{n}^{2}q^{2j}}\asymp r_{n}^{-(1+\alpha)}$, while for $\alpha>1$, $\beta_{n,q}\lesssim\max_{1\leq j\leq\ell}\frac{n^{(\alpha-1)/2(\alpha+1)}}{r_{n}^{2}q^{2j}}\asymp r_{n}^{-2}n^{\frac{\alpha-1}{2(\alpha+1)}}$. For $s_{j}\equiv s+2K\log j$, we have

[TABLE]

This shows that, for

[TABLE]

we have

[TABLE]

The first claim of the theorem follows by taking $s\asymp\log\log n$, applying Lemma A.5 and noting that $\operatorname*{\overline{lim}}_{n}\bar{\gamma}_{n}\leq\underline{\rho}^{-1}$ by the assumption. The second claim follows similarly by estimating $\beta_{n,q}$ from below, up to a multiplicative constant, by $\bar{\gamma}_{n}$ and then repeat the arguments as above. ∎

Appendix B Proof of technical lemmas

B.1. Proof of Lemma 5.1

Proof of Lemma 5.1.

The proof is a modification of that of [HW19, Proposition 2]. Let $\mathbb{M}_{n}\bm{1}_{C}\equiv\frac{2}{n}\sum_{i=1}^{n}(\bm{1}_{C}-\bm{1}_{C_{0}})(X_{i})\xi_{i}-\frac{1}{n}\sum_{i=1}^{n}(\bm{1}_{C}-\bm{1}_{C_{0}})^{2}(X_{i})$, and $M\bm{1}_{C}\equiv\mathbb{E}\left[\mathbb{M}_{n}(\bm{1}_{C})\right]=-P(\bm{1}_{C}-\bm{1}_{C_{0}})^{2}=-P\lvert C\Delta C_{0}\rvert$. Here we used the fact that $\mathbb{E}[\xi_{i}|X_{i}]=0$. Then it is easy to see that

[TABLE]

Fix $t\geq 1$. For $j\in\mathbb{N}$, let $\mathscr{C}_{j}\equiv\{C\in\mathscr{C}:2^{j-1}t\delta_{n}\leq P^{1/2}(C\Delta C_{0})<2^{j}t\delta_{n}\}$. Then by a standard peeling argument, we have

[TABLE]

Each probability term in the above display can be further bounded by

[TABLE]

By the contraction principle and moment inequality for the empirical process (cf. [GLZ00, Proposition 3.1]), we have

[TABLE]

By Chebyshev’s inequality,

[TABLE]

Under the assumption that $n\delta_{n}^{2}\geq 1\vee\lVert\xi_{1}\rVert_{2}^{2}\vee\mathbb{E}^{1/4}\max_{1\leq i\leq n}\lvert\xi_{i}\rvert^{4}$, and noting that $\phi_{n}(2^{j}t\delta_{n})\leq 2^{j}t\phi_{n}(\delta_{n})$ by the assumption that $\delta\mapsto\phi_{n}(\delta)/\delta$ is non-increasing, the right side of the above display can be further bounded up to a constant by $\sum_{j\geq 1}\left(\frac{\phi_{n}(\delta_{n})}{2^{j}t\sqrt{n}\delta_{n}^{2}}\right)^{4}+\frac{1}{t^{4}}\lesssim\frac{1}{t^{4}}$ for $t\geq 1$. The expectation bound follows by integrating the tail estimate. ∎

B.2. Proof of Lemma 5.2

Proof of Lemma 5.2.

Clearly $\mathbb{E}[\xi_{1}|X_{1}]=0$ and $\mathbb{E}[\xi_{1}^{2}|X_{1}]\lesssim 1\vee\lVert\xi_{1}\rVert_{2}^{2}$ almost surely, so we only need to verify that the assumed conditions imply (5.1) (in particular the second one). By symmetrization (cf. [GN16, Theorem 3.1.21]) and contraction principle (cf. [GN16, Theorem 3.1.17]) for the empirical process,

[TABLE]

where the last inequality follows by noting that

[TABLE]

and using triangle inequality. The proof is complete. ∎

Appendix C Some useful tools

C.1. Talagrand’s concentration inequality

We frequently use Talagrand’s concentration inequality [Tal96] in the following form given by [GN16, Theorems 3.3.9/3.3.10].

Lemma C.1**.**

Let $\mathcal{F}$ be a countable class of real-valued measurable functions such that $\sup_{f\in\mathcal{F}}\lVert f\rVert_{\infty}\leq b$. Then

[TABLE]

where $\sigma_{n}^{2}\equiv 2n^{-1/2}\mathbb{E}\sup_{f\in\mathcal{F}}\lvert\mathbb{G}_{n}(f)\rvert+\sup_{f\in\mathcal{F}}\mathrm{Var}_{P}(f)$.

C.2. Sudakov minorization

The following Sudakov minorization will be useful.

Lemma C.2** (Sudakov minorization [Sud69]).**

Let $(X_{t})_{t\in T}$ be a centered separable Gaussian process, and $\lVert t-s\rVert^{2}:=\mathbb{E}(X_{t}-X_{s})^{2}$. Then

[TABLE]

Here $C>0$ is a universal constant.

C.3. Multiplier inequalities

We use the following multiplier inequality due to [HW19, Theorem 1].

Lemma C.3**.**

Suppose that $\xi_{1},\ldots,\xi_{n}$ are i.i.d. mean-zero random variables independent of i.i.d. $X_{1},\ldots,X_{n}$. Let $\{\mathcal{F}_{k}\}_{k=1}^{n}$ be a sequence of function classes such that $\mathcal{F}_{k}\supset\mathcal{F}_{n}$ for all $1\leq k\leq n$. Assume further that there exist non-decreasing concave functions $\{\psi_{n}\}:\mathbb{R}_{\geq 0}\to\mathbb{R}_{\geq 0}$ with $\psi_{n}(0)=0$ such that

[TABLE]

holds for all $1\leq k\leq n$. Then

[TABLE]

The following alternative formulation of the multiplier inequality, proved in [HW19, Proposition 1], will also be useful.

Lemma C.4**.**

Let $\xi_{1},\ldots,\xi_{n}$ be i.i.d. symmetric mean-zero multipliers independent of i.i.d. samples $X_{1},\ldots,X_{n}$. For any function class $\mathcal{F}$,

[TABLE]

where $\lvert\xi_{(1)}\rvert\geq\cdots\geq\lvert\xi_{(n)}\rvert\geq\lvert\xi_{(n+1)}\rvert\equiv 0$ are the reversed order statistics for $\{\xi_{i}\}_{i=1}^{n}$, and $\varepsilon_{i}$’s are i.i.d. Rademacher random variables.

Acknowledgments

The major part of this work (materials presented before Section 3.3 and their proofs) is based on Chapter 4 of the author’s University of Washington Ph.D. thesis in 2018. The author would like to thank Jon Wellner and Cun-Hui Zhang for helpful discussion and encouragements. He would also like to thank four referees, an Associate Editor and the Editor for their very helpful comments and suggestions that significantly improved the quality of the paper.